Home / DOE / Analyze / Nominal Logistic Regression

Nominal Logistic Regression¶

Nominal logistic regression is the engine Run Regression uses when an output has three or more outcomes with no order between them: a colour, a day of the week, a failure mode. For an output with exactly two outcomes use Binary Logistic Regression, which is better suited to it.

How it models several outcomes¶

With \(r\) outcomes, one is taken as the baseline and the model describes each of the other \(r - 1\) against it. So there are \(r - 1\) sets of coefficients, each a comparison with the baseline rather than a description of an outcome in isolation.

For outcome \(j\) against baseline outcome \(r\):

\[ \ln\!\left(\frac{\pi_j}{\pi_r}\right) = \sum_k c_k b_{jk} \]

A coefficient is therefore the change in the log odds of landing in outcome \(j\) rather than the baseline, per unit of that term, with the other terms held fixed. Every coefficient in this report is a comparison with the baseline, which is the single most important thing to keep in mind when reading it. A coefficient does not say that outcome \(j\) becomes more likely in absolute terms, only that it becomes more likely relative to the baseline.

Transforming back, for a non-baseline outcome:

\[ \pi_j = \frac{e^{\sum_k c_k b_{jk}}}{1 + \sum_{l=1}^{r-1} e^{\sum_k c_k b_{lk}}} \]

and for the baseline itself:

\[ \pi_r = \frac{1}{1 + \sum_{l=1}^{r-1} e^{\sum_k c_k b_{lk}}} \]

The \(1\) in the denominator is the baseline's own exponent, which is 1 by construction. The probabilities across all \(r\) outcomes sum to 1.

With a design matrix of \(p\) columns the model has \((r - 1) \times p\) free parameters, so the parameter count grows with the number of outcomes. This is why a nominal response needs more runs than a quantitative one to support the same set of terms.

How the coefficients are found¶

By Newton's method, maximising the log likelihood, with no closed form.

Three limits govern the iteration, and all three are fixed in the engine. There is no control for any of them, on any dialog or on the Options page:

20 iterations.
A convergence criterion of 1e-6, which sets how small a change counts as settled.
Up to 7 half steps. Newton's method sometimes proposes a step that overshoots and makes the fit worse. When that happens the step is halved and retried, and this limits how many times.

When it does not converge¶

The message appears in red at the top of the regression table, and for a nominal fit it is that one sentence on its own. The separation sentences and the last-iteration sentence belong to the binary engine, not this one:

Model failed to converge. The results of the model are in question.

Running out of half steps is reported as a success instead, with the message Model Converged (Half stepping limit reached)., so that line is worth reading rather than skipping.

The commonest cause with a nominal response is an outcome that occurs in too few runs, or that is perfectly predicted by one factor level. An outcome observed once or twice cannot support its own set of coefficients.

What the report contains¶

Coefficients, one set per non-baseline outcome, each compared with the baseline.

Odds ratios, the exponentiated coefficients, with confidence bounds obtained by exponentiating the bounds of the coefficient:

\[ \text{OR} = e^{b_{jk}} \qquad \text{CI} = e^{\text{CI}(b_{jk})} \]

The log likelihood and its value at each iteration, with RSquaredU, AIC and BIC beside it, and the G statistic with its degrees of freedom and p value.

A Goodness of Fit Test block with two rows, Pearson and Deviance, each with its chi-square, degrees of freedom and p value. As with the binary engine, a large p value is the good outcome: it says the fitted probabilities are consistent with the observed counts.

A predicted probability table, two columns wide with one row per outcome, the baseline included. The baseline's cell is not a subtraction from the others: it holds the reference-level formula directly, which is what makes the column of probabilities total exactly 1.

Which outcome is the baseline¶

The baseline is the outcome with no coefficients of its own. It is not left out of the model: every other outcome's coefficients are measured against it, so it is the reference the whole report is built on.

This matters when reading the report, because the same data with a different baseline gives different coefficients describing the same fit. The predicted probabilities do not change.

Predictions¶

For a nominal output with \(L\) non-reference levels there is one sum per level, \(\eta_k = \sum_j c_j b_{jk}\), and the probability of level \(k\) is

\[ p_k = \frac{e^{\eta_k}}{1 + \sum_{l=1}^{L} e^{\eta_l}} \]

The \(1\) in the denominator is the reference level, whose \(e^{\eta}\) is \(1\) by construction. So the reference level's own probability is

\[ p_{\text{ref}} = \frac{1}{1 + \sum_{l=1}^{L} e^{\eta_l}} \]

and the probabilities across all levels sum to \(1\).

Full details: Prediction Equation.

When charting a nominal output you choose which outcome's probability to plot, and each outcome gets its own chart, because there is no single response value to draw.

References¶

Agresti, A. (2013). Categorical Data Analysis, 3rd edition. Wiley.
Hosmer, D. W., Lemeshow, S., and Sturdivant, R. X. (2013). Applied Logistic Regression, 3rd edition. Wiley.
McCullagh, P., and Nelder, J. A. (1989). Generalized Linear Models, 2nd edition. Chapman and Hall.