Home / Statistical Tools / MSA / Attribute MSA: Crosstabulations Method / Math Details

Math Details¶

Every statistic on an Attribute MSA report, and the rule that decides when it is reported at all.

An attribute study has no variance components, so none of the MSA Math pages apply to it. Everything it computes is on this page.

Notation¶

Term	Description
\(N\)	the number of parts
\(n\)	the number of appraisers
\(m\)	the number of trials each appraiser gave each part
\(L\)	the number of distinct levels in the ratings and the standard together
\(\alpha\)	Alpha:, default \(0.05\)
\(C(x,2)\)	the number of unordered pairs that can be drawn from \(x\) things, \(\dfrac{x(x-1)}{2}\)
\(\Phi\)	the standard normal distribution function

Every proportion below is stored as a fraction between 0 and 1 and displayed by Excel as a percentage, so a stored \(0.8400\) reads as \(84.00\%\).

How levels become codes¶

The levels are the distinct values found in the Rating columns and the Standard column together, compared without regard to case. The first spelling met wins, and a later differently cased spelling is folded into it and reported as a note.

Those levels are then put in order and numbered \(0, 1, \dots, L-1\):

If every level reads as a number, they are ordered by numeric value, ascending. Equal values spelled differently keep the order they were met in.
Otherwise, at Ordinal, the order you set in the Levels block is used, if it covers every level found.
Otherwise they are ordered alphabetically, ignoring case.

Rule 3 is the ordinary default and is silent. A note is printed on the sheet only in the one case where an order really was lost: an Ordinal study that had a saved level order which no longer covers the levels found in the data, so the alphabetical order had to be used instead. Never having set an order is not that case.

For an ordinal study the code is the rank: a higher code is a higher level, and the Kendall statistics below read the codes as ranks on that basis.

Assessment agreement¶

The four Assessment Agreement tables all report the same proportion, and differ only in what has to match for a part to count:

\[ \text{Percent} = \frac{\text{\# Matched}}{\text{\# Inspected}} , \qquad \text{\# Inspected} = N \]

The denominator is always the number of parts, never the number of ratings. The numerator is all-or-nothing per part, which is what makes this the strictest figure on the report:

Section	A part is matched when
Within Appraiser	that appraiser gave the same level on all \(m\) trials
Each Appraiser vs Standard	all \(m\) of that appraiser's levels equal the standard
Between Appraisers	every one of the \(n \times m\) ratings of the part is the same level
All Appraisers vs Standard	every one of the \(n \times m\) ratings equals the standard

One disagreeing trial out of twelve loses the whole part. That is the intent: a system that is right eleven times in twelve on a part has still not decided what the part is.

The confidence interval¶

Every agreement percentage and every effectiveness percentage carries an exact Clopper and Pearson binomial interval, computed through the F distribution. With \(x\) matched out of \(N\) inspected:

\[ \text{Lower} = \frac{v_1 u}{v_2 + v_1 u}, \quad v_1 = 2x, \quad v_2 = 2(N - x + 1), \quad u = F^{-1}_{v_1, v_2}\!\left(\tfrac{\alpha}{2}\right) \]

\[ \text{Upper} = \frac{v_3 w}{v_4 + v_3 w}, \quad v_3 = 2(x+1), \quad v_4 = 2(N - x), \quad w = F^{-1}_{v_3, v_4}\!\left(1 - \tfrac{\alpha}{2}\right) \]

At the two boundaries the interval becomes one-sided, and the whole of \(\alpha\) goes into the one bound that exists rather than half of it:

\[ x = 0: \quad \text{Lower} = 0, \quad \text{Upper} = 1 - \alpha^{1/N} \]

\[ x = N: \quad \text{Lower} = \alpha^{1/N}, \quad \text{Upper} = 1 \]

This is why perfect agreement does not produce an interval of zero width. Twenty parts all matched gives a lower bound of \(0.05^{1/20} = 0.8609\), not \(1\).

The Pairwise % Agreement, Conformance and Assessment Disagreement figures carry no interval.

Fleiss' kappa¶

Kappa asks how much of the agreement is more than you would expect from the level frequencies alone. It is reported for every section, one row per level plus an Overall row.

Let \(x_{ij}\) be the number of raters who put part \(i\) at level \(j\), and \(r\) the number of raters. What counts as a rater differs by section, and this is the part most easily got wrong:

Section	\(r\)
Within Appraiser	\(m\), that appraiser's own trials
Between Appraisers	\(n \times m\), every appraiser and trial as its own rater
the two vs Standard sections	\(2\) per comparison, the rating column and the standard, then averaged

For one level \(j\):

\[ p_j = \frac{1}{Nr}\sum_{i} x_{ij} \]

\[ P_{o,j} = \frac{\sum_i \left[x_{ij}^2 + (r - x_{ij})^2\right] - Nr}{Nr(r-1)} , \qquad P_{e,j} = p_j^2 + (1 - p_j)^2 \]

\[ \kappa_j = \frac{P_{o,j} - P_{e,j}}{1 - P_{e,j}} , \qquad \operatorname{Var}(\kappa_j) = \frac{2}{Nr(r-1)} \]

Overall, across the levels present:

\[ P_o = \frac{\sum_i \sum_j x_{ij}^2 - Nr}{Nr(r-1)} , \qquad p_j = \frac{\sum_i x_{ij}}{Nr} , \qquad P_e = \sum_j p_j^2 \]

\[ \kappa = \frac{P_o - P_e}{1 - P_e} \]

\[ \operatorname{Var}(\kappa) = \frac{2}{Nr(r-1)S^2}\left(S^2 - \sum_j p_j(1-p_j)(1-2p_j)\right) , \qquad S = \sum_j p_j(1 - p_j) \]

Against a standard, kappa is computed once per comparison, treating that rating column and the standard as two raters, and the \(g\) results are averaged:

\[ \bar{\kappa} = \frac{1}{g}\sum_{c=1}^{g}\kappa_c , \qquad \operatorname{Var}(\bar{\kappa}) = \frac{1}{g^2}\sum_{c=1}^{g}\operatorname{Var}(\kappa_c) \]

with \(g = m\) for Each Appraiser vs Standard and \(g = n \times m\) for All Appraisers vs Standard. A comparison that could not produce a kappa is dropped and \(g\) reduced with it.

Cohen's kappa¶

Where a section compares exactly two things, a second kappa is reported alongside Fleiss'. From the \(L \times L\) table of proportions \(p_{ij}\), with row margins \(p_{i+}\) and column margins \(p_{+i}\):

\[ P_o = \sum_i p_{ii} , \qquad P_e = \sum_i p_{i+}\,p_{+i} , \qquad \kappa = \frac{P_o - P_e}{1 - P_e} \]

\[ SE(\kappa) = \frac{\sqrt{P_e + P_e^2 - \sum_i p_{i+}p_{+i}\left(p_{i+} + p_{+i}\right)}}{(1 - P_e)\sqrt{N}} \]

That is the standard error under the hypothesis that kappa is zero, which is the right one for the test below and the wrong one for building an interval around kappa.

A per-level Cohen row collapses the table to that level against everything else and applies the same formula to the resulting two by two table.

Cohen's kappa is reported only where the comparison really is between two things: Within Appraiser with exactly two trials, Between Appraisers with exactly two appraisers giving one trial each, and against a standard. Elsewhere the heading appears with a sentence saying which shape it needs.

The kappa test¶

Both kappas are tested the same way, one-sided, against the hypothesis that there is no agreement beyond chance:

\[ Z = \frac{\kappa}{SE(\kappa)} , \qquad p = 1 - \Phi(Z) \]

The column is headed P(vs > 0) to say what is being tested. Neither \(Z\) nor \(p\) is computed when the standard error is not positive.

When kappa is not available¶

\(\kappa\) needs \(1 - P_e > 0\). Expected agreement of exactly 1, which happens when both columns hold one single level throughout, leaves nothing for kappa to divide by, and the row reads N/A rather than 1. Perfect observed agreement with \(P_e < 1\) is an ordinary \(\kappa = 1\) and is reported normally.

Kappa between appraiser pairs¶

Agreement Between Appraiser Pairs takes each pair of appraisers, stacks every trial of both into one table of \(N \times m\) paired ratings, and reports Cohen's kappa on it. The point estimate is the formula above; the standard error is not. Here the general large-sample form is used, the one that describes the spread of kappa itself rather than its spread under a null hypothesis:

\[ \operatorname{Var}(\kappa) = \frac{\displaystyle\sum_i p_{ii}\left[1 - (p_{i+} + p_{+i})(1-\kappa)\right]^2 + (1-\kappa)^2\sum_{i \ne j} p_{ij}\left(p_{+i} + p_{j+}\right)^2 - \left[\kappa - P_e(1-\kappa)\right]^2}{N\,(1 - P_e)^2} \]

No \(Z\) and no \(p\) are printed for this table. The two standard errors answer different questions, and running the null hypothesis test on the general standard error would be the wrong test.

Ratings are paired by trial in worksheet row order, and the report says so in a note under the table.

Pairwise percent agreement¶

A gentler figure than assessment agreement, and it is on the report because the strict one loses a whole part to a single disagreement. Instead of demanding unanimity, it counts pairs of ratings that agree.

For part \(i\) holding \(N_i = n \times m\) ratings, of which \(x_{il}\) are at level \(l\):

\[ \text{By Part}_i = \frac{\sum_l C(x_{il},\,2)}{C(N_i,\,2)} \]

For appraiser \(j\), counting only the pairs that appraiser is part of:

\[ \text{By Appraiser}_j = \frac{\sum_i \sum_l \left[C(a_{ijl},2) + a_{ijl}\left(x_{il} - a_{ijl}\right)\right]}{\sum_i \left[C(m,2) + m\left(N_i - m\right)\right]} \]

where \(a_{ijl}\) is the number of appraiser \(j\)'s own trials on part \(i\) at level \(l\). The numerator's two terms are the pairs inside that appraiser's own trials and the pairs between their trials and everybody else's.

\[ \text{Overall} = \frac{\sum_i \sum_l C(x_{il},2)}{\sum_i C(N_i,2)} \]

A cell with no pairs to compare, which happens only with one appraiser giving one trial, reads N/A.

Effectiveness¶

Where assessment agreement scores a part, effectiveness scores a decision. Each individual rating is right or wrong on its own:

\[ \text{Effectiveness}_j = \frac{\text{\# Correct}_j}{\text{\# Decisions}_j} = \frac{\text{\# Correct}_j}{N \times m} \]

and the Overall row pools every appraiser, over \(N \times m \times n\) decisions. Both carry the Clopper and Pearson interval above.

An appraiser who gets eleven of twelve trials right on every part scores near 1 here and 0 on assessment agreement. Reading the two together is the point of having both.

Assessment disagreement¶

Two-level data only, and only against a standard. With level 0 the lower code and level 1 the higher:

\[ \#(1 \mid 0) = \text{parts the appraiser called level 1 on every trial whose standard is level 0} \]

\[ \%(1 \mid 0) = \frac{\#(1 \mid 0)}{\text{parts whose standard is level 0}} \]

and symmetrically for \(\#(0 \mid 1)\). The third pair counts the parts the appraiser could not decide about:

\[ \#\text{Mixed} = \text{parts whose trials by this appraiser disagreed with each other} , \qquad \%\text{Mixed} = \frac{\#\text{Mixed}}{N} \]

Each of the three denominators is different, which the notes under the table spell out. A part counted as Mixed is not counted in either of the other two, because the appraiser did not consistently call it anything.

Conformance¶

Reported when the Standard column holds exactly two distinct values and a conforming level has been named. The number of rating levels is not part of that rule, only the standard's own values.

\[ P(\text{False Alarm})_j = \frac{\text{ratings by } j \text{ of a conforming part that were not the conforming level}}{\text{conforming parts} \times m} \]

\[ P(\text{Miss})_j = \frac{\text{ratings by } j \text{ of a nonconforming part that were not the nonconforming level}}{\text{nonconforming parts} \times m} \]

Both denominators count assessments, parts in that state times trials, and both are per appraiser. There is no pooled row.

A false alarm scraps a good part; a miss ships a bad one. They are reported separately because they almost never cost the same.

Kendall's statistics¶

Ordinal data only, and the level codes are the ranks.

Coefficient of concordance¶

Reported for Within Appraiser and Between Appraisers. Tied parts share the mean of the rank positions they occupy, and each rank column carries a tie correction \(T_j = \sum (t^3 - t)\) over its tie groups of size \(t\). With \(c\) rank columns and \(R_i\) the rank sum of part \(i\) across them:

\[ W = \frac{12\sum_i R_i^2 - 3c^2N(N+1)^2}{c^2N(N^2-1) - c\sum_j T_j} \]

\[ \chi^2 = c(N-1)W , \qquad \nu = N - 1 , \qquad p = 1 - F_{\chi^2,\,\nu}\!\left(\chi^2\right) \]

\(c = m\) within an appraiser and \(c = n \times m\) between appraisers. \(W\) reads N/A when the denominator is not positive, which means every column was wholly tied.

Correlation coefficient¶

Reported for the two vs Standard sections. Between one rank column and the standard, over all \(C(N,2)\) pairs of parts:

\[ \tau_b = \frac{C - D}{\sqrt{\left(\text{pairs} - \text{tied on A}\right)\left(\text{pairs} - \text{tied on B}\right)}} \]

with \(C\) concordant pairs and \(D\) discordant. The \(g\) comparisons are then averaged:

\[ \text{Coef} = \frac{1}{g}\sum_{c=1}^{g}\tau_{b,c} , \qquad SE(\text{Coef}) = \sqrt{\frac{1}{g} \cdot \frac{2(2N+5)}{9N(N-1)}} \]

\[ S = g \cdot \text{Coef} \cdot \frac{N(N-1)}{2} , \qquad \operatorname{Var}(S) = \frac{N(N-1)(2N+5)}{18} \]

\[ Z = \frac{S - \operatorname{sign}(S)}{\sqrt{g\operatorname{Var}(S)}} , \qquad p = 2\left(1 - \Phi\!\left(|Z|\right)\right) \]

The \(-\operatorname{sign}(S)\) is a continuity correction, always towards zero. At \(S\) exactly zero nothing is subtracted, so \(Z = 0\) and \(p = 1\). This test is two-sided, unlike the one-sided kappa test above.

The tie correction lives in \(\tau_b\) alone, in the two terms subtracted from the denominator. It does not enter the standard error or \(Z\).

Which statistics appear for which data type¶

Statistic	Binomial	Nominal	Ordinal
Assessment agreement and its interval	Yes	Yes	Yes
Fleiss' kappa	Yes	Yes	Yes
Cohen's kappa	Yes	Yes	Yes
Kappa between appraiser pairs	Yes	Yes	Yes
Pairwise percent agreement	Yes	Yes	Yes
Effectiveness, agreement counts, misclassifications	Yes	Yes	Yes
Conformance	Yes	Yes	Yes
Assessment disagreement	when the data holds exactly two levels	when the data holds exactly two levels	No
Kendall's coefficient of concordance	No	No	Yes
Kendall's correlation coefficient	No	No	Yes

Ordinal is the only choice that changes what is computed. Binomial and Nominal are the same study: the engine is told nominal for both, and the assessment disagreement row above is decided by the number of levels in the data, not by which of the two you picked. Ordinal data is excluded from that row because it must hold three or more levels to run at all.

A row marked Yes still needs its section's own requirement met. The rules are on Options, and a statistic whose requirement is not met is left out in silence rather than reported as zero.

Degenerate cases¶

The study is deliberately tolerant. Most shortfalls omit a table rather than stopping the run.

Situation	What happens
One appraiser	Between Appraisers, All Appraisers vs Standard and the appraiser pair table are omitted
One trial per appraiser	Within Appraiser and everything under it is omitted
No standard column	Every comparison against a standard is omitted, including conformance, effectiveness and the Kendall correlation
Every rating and the standard at one level	The study runs. Kappa rows read N/A, the Cohen tables are not printed, and there is no disagreement or conformance table
Both appraisers of a pair gave one level throughout, and the same level	Kappa and its standard error both read N/A, because expected agreement is exactly 1
Exactly one appraiser of a pair gave one level throughout, or both did at different levels	Kappa is exactly 0, its standard error reads N/A
Ordinal on fewer than three levels	The study is refused, with a sentence saying so

References¶

Clopper, C.J. and Pearson, E.S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26(4), 404-413.
Johnson, N.L. and Kotz, S. (1969). Discrete Distributions. Wiley.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1), 37-46.
Fleiss, J.L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin 76(5), 378-382.
Fleiss, J.L., Cohen, J. and Everitt, B.S. (1969). Large sample standard errors of kappa and weighted kappa. Psychological Bulletin 72(5), 323-327.
Kendall, M.G. and Babington Smith, B. (1939). The problem of m rankings. Annals of Mathematical Statistics 10(3), 275-287.
Kendall, M.G. (1945). The treatment of ties in ranking problems. Biometrika 33(3), 239-251.
Automotive Industry Action Group (2010). Measurement Systems Analysis Reference Manual, 4th edition. AIAG.