Home / DOE / Analyze / Binary Logistic Regression

Binary Logistic Regression¶

Binary logistic regression is the engine Run Regression uses when an output has exactly two outcomes: pass and fail, present and absent, 0 and 1. For an output with three or more outcomes, see Nominal Logistic Regression.

Why not least squares¶

Least squares would happily predict a probability of 1.4 or of \(-0.2\), which are not probabilities. Logistic regression fits on a scale where that cannot happen, by modelling the log odds of the outcome as linear in the factors and then transforming back:

\[ \ln\!\left(\frac{\pi}{1 - \pi}\right) = \sum_j c_j b_j \qquad \Longleftrightarrow \qquad \pi = \frac{1}{1 + e^{-\sum_j c_j b_j}} \]

where \(\pi\) is the predicted probability, \(c_j\) the coded value of term \(j\) and \(b_j\) its coefficient. The right-hand form is an S curve that approaches 0 and 1 without reaching either, so every prediction is a valid probability.

How the coefficients are found¶

There is no closed form. The coefficients are found by iteration, maximising the log likelihood:

\[ \ln L = \sum_i \left[ y_i \ln \pi_i + (1 - y_i)\ln(1 - \pi_i) \right] \]

Quantum XL uses iteratively reweighted least squares, a form of Newton's method: it starts from a guess, computes the likelihood, takes a step, and repeats until the coefficients stop changing. The log likelihood at every iteration is kept, so the report can show how the fit converged.

The iteration limits are fixed and there is no control for them. The fit is allowed 20 iterations, with a convergence criterion of 1e-6 and up to 7 half steps. None of the three appears on any dialog or on the Options page.

When it does not converge¶

Iteration is not guaranteed to succeed, and Quantum XL says so plainly rather than presenting a doubtful model as a good one. The message appears in red at the top of the regression table, and it is one message built up from sentences, not several lines:

Model failed to converge. The results of the model are in question. Detected complete separation of data points. Results are based on last iteration; validity of the model is in question.

The middle sentence is present only when separation was detected, and reads Detected quasicomplete separation of data points. for the partial case. The last sentence is always appended when the fit did not converge.

A third outcome is reported as a success. If the fit runs out of half steps, the message is Model Converged (Half stepping limit reached). and the model is treated as converged, so this line is worth reading rather than skipping.

The usual cause of a failure is separation: the factors predict the outcome too perfectly. If every run above some temperature passed and every run below it failed, the best fit is an infinitely steep step, and no finite coefficient reaches it, so the iteration walks off toward infinity instead of settling.

Quantum XL detects this and distinguishes two degrees of it:

Complete separation, where every run is predicted almost perfectly. The factors separate the outcomes with no overlap at all.
Partial, or quasi-complete, separation, where most runs are predicted almost perfectly. Usually one factor, or one level of one, splits the outcome cleanly.

Separation is not a software failure and not bad data. It means the data pin the outcome down so completely that no finite coefficient fits it, which is why the fit reports rather than returns an answer.

What the report contains¶

Beyond coefficients and their standard errors:

Odds ratios, with lower and upper confidence bounds. The odds ratio is the exponentiated coefficient:

\[ \text{OR} = e^{b_j} \]

so it says how the odds of the outcome multiply per unit of the factor. An odds ratio of 1 means no effect, because multiplying odds by 1 leaves them unchanged.

The log likelihood of the fitted model, and its value at each iteration.

The G statistic, which tests the whole model in the way F tests a least squares model:

\[ G = -2\left(\ln L_{\text{constant only}} - \ln L_{\text{full model}}\right) \]

compared against a chi-square distribution on \(p - 1\) degrees of freedom, where \(p\) is the number of parameters. A small p value on G says the factors together explain something.

Goodness of fit, as a Pearson chi-square and a deviance chi-square with their degrees of freedom and p values. Here a large p value is the good outcome: it says the fitted probabilities are consistent with the observed counts.

Hosmer-Lemeshow is not reported

The Hosmer-Lemeshow goodness of fit test is deliberately not included, because its result depends on how the data is grouped into bins and there is no single agreed way to do that, so the same data can pass or fail depending on a choice the test does not fix. The Pearson and deviance statistics are reported instead.

Residual diagnostics specific to logistic models: delta chi-square, delta deviance, delta beta and its standardized form, each measuring what happens to the fit if one run is removed. See Residual Plots and Binomial Logistic Residuals.

Predictions¶

For a binary output the same sum is formed first, then passed through the logistic function to give a probability:

\[ p = \frac{1}{1 + e^{-\sum_j c_j b_j}} \]

That is the probability of the first level. The probability of the second is \(1 - p\).

Full details: Prediction Equation.

Charts of a binary output plot a probability rather than a response value, which is worth remembering when reading a surface or an interaction plot: the vertical axis runs from 0 to 1 and the surface flattens near both ends because the logistic curve does.

References¶

Hosmer, D. W., Lemeshow, S., and Sturdivant, R. X. (2013). Applied Logistic Regression, 3rd edition. Wiley.
Agresti, A. (2013). Categorical Data Analysis, 3rd edition. Wiley.
Albert, A., and Anderson, J. A. (1984). On the Existence of Maximum Likelihood Estimates in Logistic Regression Models. Biometrika, 71(1), 1 to 10.