Home / DOE / Analyze / Binary Logistic Regression / Results

Regression Results for Binary Logistic Regression¶

Run Regression fits every binary output with Binary Logistic Regression and writes the results to a regression sheet. This page explains each part of that report.

A logistic report has more model-level statistics than a least squares one, because there is no single R squared that settles the question, so several partial answers are given instead.

The coefficient table¶

Coeff¶

The coefficient, in the units of the model, normally coded. On the logistic scale a coefficient is a change in log odds rather than a change in the response, so it cannot be read directly as an effect on the probability. Use the odds ratio for a multiplicative reading, or the prediction block for probabilities.

SE¶

How much the coefficient would move if the experiment were repeated. Its main use is as the denominator of the Z statistic.

Z¶

The coefficient divided by its standard error:

\[ Z = \frac{b_j}{SE(b_j)} \]

A signal to noise ratio for one term. Logistic regression uses a normal reference distribution rather than a t distribution, which is why this column is Z and not T.

P¶

The 2-tail p value from comparing Z against the normal distribution. Coloured as elsewhere in Quantum XL:

Colour	p value
Red	0.05 or below
Blue	above 0.05, up to and including 0.1
Black	above 0.1

Odds Ratio, 95% CI Lower, 95% CI Upper¶

The odds of an outcome are its probability divided by the probability of it not happening:

\[ \text{odds} = \frac{\pi}{1 - \pi} \]

The odds ratio is the exponentiated coefficient, so it says how the odds multiply per unit of the term:

\[ \text{OR} = e^{b_j} \]

An odds ratio of 1 means no effect. The confidence bounds are the exponentiated bounds of the coefficient, so they are not symmetric about the odds ratio even though the coefficient's bounds are symmetric about it.

A confidence bound on a coefficient is the coefficient plus or minus a multiplier times its standard error. The binary engine uses a fixed normal multiplier rather than a \(t\) value. It multiplies by the literal constant \(1.95996398454005\), the 95 percent two-sided standard normal value:

\[ b_j \pm 1.95996398454005 \times SE(b_j) \]

That number does not change with the number of runs or the number of terms. A bound recomputed with a \(t\) multiplier will therefore not reproduce the product's number.

The coefficient bounds themselves are not written to a sheet. What the sheet reports is the odds ratio bounds, which are these bounds exponentiated:

\[ \left(e^{\,b_j - 1.95996398454005\, SE(b_j)},\; e^{\,b_j + 1.95996398454005\, SE(b_j)}\right) \]

Because the exponential is not linear, the odds ratio bounds are not symmetric about the odds ratio, even though the coefficient bounds are symmetric about the coefficient.

Full details: Regression Diagnostics.

Odds are not probabilities, and the difference is large

Suppose 90 percent of one group and 20 percent of another do something. One group is \(0.9 / 0.2 = 4.5\) times as likely. But the odds are \(0.9/0.1 = 9\) against \(0.2/0.8 = 0.25\), an odds ratio of 36. Both numbers are correct and they describe the same data.

The prediction block reports probabilities; the coefficient table's odds ratio columns report odds. They are different quantities computed from the same fit.

In Model¶

A checkbox per term. Clearing it and re-running the regression drops that term. It also removes the term from every chart and prediction.

Model fit statistics¶

LogLikelihood¶

The log of the likelihood the engine reports, which is the last entry of its log-likelihood history, the quantity the iteration maximised. The engine sums over the \(J\) distinct covariate patterns, each carrying its own trial count \(m_i\) and event count \(y_i\):

\[ \ln L = \sum_{i=1}^{J} \left[ y_i \ln \hat{\pi}_i + (m_i - y_i) \ln (1 - \hat{\pi}_i) \right] \]

When every pattern holds a single trial, which is what a design with one unit per run gives, this reduces to the familiar form summed over runs, \(y_i \ln \hat{\pi}_i + (1 - y_i)\ln(1 - \hat{\pi}_i)\).

It is always negative and has no absolute interpretation. It exists to be compared: between nested models, and inside the statistics below.

RSquaredU¶

McFadden's pseudo R squared, which compares the fitted model's likelihood with that of a model containing only a constant:

\[ R^2_U = 1 - \frac{\ln L_{\text{full}}}{\ln L_{\text{constant only}}} \]

It runs from 0 to 1, but it is not the proportion of variation explained and its values are not comparable with a least squares R squared. Quantum XL computes it and writes it out; it applies no threshold to it.

AIC and BIC¶

Both trade fit against complexity, and for both lower is better:

\[ \text{AIC} = -2\ln L + 2k \qquad \text{BIC} = -2\ln L + k \ln n \]

with \(k\) terms and \(n\) observations. BIC charges more per term than AIC whenever \(\ln n > 2\), so it prefers smaller models on all but the smallest experiments.

Neither is meaningful alone. They exist to compare two models fitted to the same data, and the one with the lower value is preferred.

This is the plain AIC, not the small sample corrected AICc.

RMSE¶

Root mean square error, the typical distance between an observed outcome and its predicted probability. The engine again sums over the \(J\) covariate patterns rather than over runs, accumulating one term of \((1 - \hat{\pi}_i)^2\) for each of a pattern's \(y_i\) events and one term of \(\hat{\pi}_i^2\) for each of its \(m_i - y_i\) non-events, then divides by the total sample size \(n = \sum_i m_i\):

\[ \text{RMSE} = \sqrt{\frac{\sum_{i=1}^{J} \left[ y_i (1 - \hat{\pi}_i)^2 + (m_i - y_i)\hat{\pi}_i^2 \right]}{n}} \]

When every pattern holds a single trial, this reduces to the familiar form summed over runs, \(\sqrt{\sum_i (y_i - \hat{\pi}_i)^2 / n}\).

Testing the model as a whole¶

G Stat, G df, G P-Value¶

G tests whether the terms together explain anything, the logistic counterpart of the F test:

\[ G = -2\left(\ln L_{\text{constant only}} - \ln L_{\text{full}}\right) \]

compared against a chi-square distribution on G df degrees of freedom. A G P-Value below 0.05 says you can be at least 95 percent confident that not all the slopes are zero, that is, the model predicts better than the constant alone.

Goodness of Fit Test¶

One block heading, Goodness of Fit Test, over two rows labelled Pearson and Deviance, with the columns Chi-Sq, DF and P.

Pearson¶

The sum of squared Pearson residuals, compared against chi-square. Here a small p value is the bad outcome: below 0.05 indicates significant lack of fit, meaning the fitted probabilities do not match the observed counts.

This is the reverse of the G test, and mixing them up is the commonest error in reading a logistic report. G small is good; goodness of fit small is bad.

The Pearson statistic compares observed counts with the counts the fitted model expects, and it is referred to a chi-square distribution. The binary engine computes the statistic and chooses the degrees of freedom itself, and asks the regression library for the upper tail probability of the chi-square distribution at that statistic on those degrees of freedom.

It is the sum of the squared Pearson residuals over the \(J\) covariate patterns:

\[ \chi^2_P = \sum_{i=1}^{J} r_{P,i}^{\,2} \]

on \(J - p\) degrees of freedom, the number of patterns less the number of model terms counting the constant. A pattern whose residual is not a number is left out of the sum rather than making the whole sum not a number. That arises at a pattern the model reproduces exactly, where the fitted probability is exactly \(0\) or exactly \(1\) and the observed count agrees with it, so the residual is zero divided by zero. When the number of patterns equals the number of terms there are no degrees of freedom left and no p value is reported.

How much replication a design has therefore matters, because \(J\) enters the degrees of freedom directly. A design in which no set of factor settings is repeated has \(J\) equal to its number of runs.

Deviance¶

The same question by way of the deviance. Again a p value below 0.05 indicates lack of fit.

The deviance test needs replication to be meaningful. When the number of distinct covariate patterns approaches the number of observations it becomes unreliable, because each pattern has too few observations to say anything about fit. With several runs per covariate pattern it becomes useful.

The deviance statistic asks the same question as the Pearson statistic by a different route, comparing the fitted model's log-likelihood with that of a model that reproduces every observed proportion exactly. It is referred to the same chi-square distribution on the same degrees of freedom as the Pearson statistic, and again the binary engine forms the statistic and chooses the degrees of freedom while the regression library supplies the upper tail probability.

It is the sum of the squared deviance residuals over the \(J\) covariate patterns:

\[ \chi^2_D = \sum_{i=1}^{J} d_i^{\,2} \]

on \(J - p\) degrees of freedom, the same divisor the Pearson statistic uses. The sign each residual carries is lost in the squaring, so the two statistics share their degrees of freedom but not their arithmetic and they do not have to agree.

Full details: Logistic Estimation.

Hosmer-Lemeshow is not reported

Its result depends on how observations are grouped into bins, and there is no single agreed way to do that, so the same data can pass or fail on a choice the test leaves open. Pearson and deviance are reported instead.

Association, and the Summary Measures¶

The sheet carries these as two blocks, headed Association and Summary Measures. They ask a different question from the tests above: not whether the model fits, but whether it ranks correctly. If the model gives a higher probability to the run that actually had the higher outcome, it is ordering the data properly even if the probabilities themselves are off.

Only pairs made of one observed event and one observed non-event are counted. Two events, or two non-events, are never classified at all, because there is no correct order for them. Each such pair is then sorted by the two runs' predicted probabilities, not by their outcomes:

Row on the sheet	When a pair counts here
Concordant	the event has the higher predicted probability
Discordant	the event has the lower predicted probability
Ties	the two predicted probabilities fall in the same 0.002-wide band. This includes both members of any pair drawn from a single covariate pattern, since those share one predicted probability

The 0.002 band is a fixed grid, not a tolerance around each pair. Quantum XL divides each probability by 0.002 and keeps the whole number, so a pair is tied when both probabilities land in the same slot. Two probabilities either side of a slot boundary are not a tie however close they are: 0.1999 and 0.2001 are 0.0002 apart and count as discordant or concordant, while 0.2001 and 0.2019 are nearly ten times further apart and count as a tie. The width is not a setting.

The report gives the count and the percentage of each row. The counts are then summarised three ways, each running from \(-1\) to \(+1\).

Writing \(C\), \(D\) and \(T\) for the concordant, discordant and tied counts:

Somers' D¶

\[ D_{\text{Somers}} = \frac{C - D}{C + D + T} \]

Ties are in the denominator, so tied pairs dilute the result: a model with many ties cannot reach 1 even if it ranks every non-tied pair correctly.

Goodman-Kruskal Gamma¶

\[ \gamma = \frac{C - D}{C + D} \]

Ties are excluded, so this asks only about pairs the model could have ranked. It is therefore never smaller in magnitude than Somers' D on the same data.

The difference between these two is exactly the treatment of ties, and it is worth knowing which you are quoting.

Kendall's Tau¶

\[ \tau_a = \frac{C - D}{\tfrac{1}{2} n (n - 1)} \]

The sheet's row is labelled Kendall's Tau; the quantity computed is tau-a. Its denominator is every possible pair, whether the model could rank it or not, so it is usually the smallest of the three.

The prediction block¶

At the top of the table, this is where to go for probabilities rather than odds. Enter factor settings and read back the predicted probability of the outcome.

For a binary output the same sum is formed first, then passed through the logistic function to give a probability:

\[ p = \frac{1}{1 + e^{-\sum_j c_j b_j}} \]

That is the probability of the first level. The probability of the second is \(1 - p\).

Full details: Prediction Equation.

The convergence line above the table¶

There is always a line in that cell. On a fit that converged it reads Binary Logistic Output - Converged in n iterations., where n counts the entries in the log likelihood history, which includes the entry made before the first iteration.

If the model did not converge, the line is a warning in red instead, and the coefficients shown are from the last iteration rather than from a converged fit. Every statistic on this page is then unreliable. See Binary Logistic Regression for what causes it and what to do.

References¶

Hosmer, D. W., Lemeshow, S., and Sturdivant, R. X. (2013). Applied Logistic Regression, 3rd edition. Wiley.
Agresti, A. (2013). Categorical Data Analysis, 3rd edition. Wiley.
McFadden, D. (1974). Conditional Logit Analysis of Qualitative Choice Behavior. In Frontiers in Econometrics, Academic Press.
Somers, R. H. (1962). A New Asymmetric Measure of Association for Ordinal Variables. American Sociological Review, 27(6), 799 to 811.