Home / DOE / Analyze / Ordinary Least Squares Regression
Ordinary Least Squares Regression¶
Ordinary least squares is the engine Quantum XL uses for a quantitative output. It chooses the coefficients that make the sum of the squared differences between the measured responses and the model's predictions as small as possible.
You do not select it. Run Regression uses it automatically for any output whose type is Quantitative.
The fit¶
Writing \(X\) for the design matrix, which holds a column for the constant and a column for each term, \(y\) for the vector of responses, and \(b\) for the coefficients:
With weights, where \(W\) is a diagonal matrix carrying one weight per run:
Weights are used for the response model when the design carries a weights column, so a row standing for more observations counts for more. The standard deviation model is always fitted unweighted, whatever Use weighted S-Hat regression on the Regression page of DOE Options is set to. See The S-Hat Model.
What the design has to support¶
\((X'X)^{-1}\) exists only when the columns of \(X\) are independent. If they are not, the run stops and reports that the X matrix is rank deficient, which means these runs cannot tell two or more terms apart. It is a property of the design and the requested model together, not an error in the data.
Coding affects the numbers in \(X\) and therefore the coefficients, but not which model is fitted or what it predicts.
Every quantitative factor is put on a common scale before the regression runs. One expression does it, whichever coding is in force, using a centre \(m_c\) and a scale \(s_c\) that the coding chooses:
A categorical factor is never coded this way. Its coded value is \(1\), and the level it sits at selects which coefficient is used instead.
An interaction's coded value is the product of the coded values of the factors in it.
Full details: Design Coding.
What comes with the fit¶
Least squares gives more than the coefficients. The report also carries, for each term, a standard error, a t statistic, a p value and a variance inflation factor; for the model, R squared and its adjusted form, an F statistic and its significance; and for each run, a fitted value, a residual in several scaled forms, a leverage and a Cook's distance.
Where the design has replicated settings it also separates the residual variation into pure error and lack of fit, which is the one part of the report that can say the model is the wrong shape rather than merely imprecise.
All of it is described on Regression Results for OLS, and the mathematics is on Sums of Squares.
What least squares assumes¶
The fit itself needs nothing but independent columns. The standard errors, t statistics and p values need more: that the residuals are independent of each other, have constant spread across the range of the fitted values, and are roughly normal.
Those assumptions are not checked automatically, which is why the diagnostics exist. Residual Plots tests them directly, and it is worth running before quoting a p value from this report.
See Also¶
References¶
- Draper, N. R., and Smith, H. (1998). Applied Regression Analysis, 3rd edition. Wiley.
- Montgomery, D. C. (2013). Design and Analysis of Experiments, 8th edition. Wiley.