Agreement and kappa

How LangChef decides whether your judge can be trusted, worked from real numbers, with the code and the papers behind each step. No statistics background assumed.

                judge fail   judge pass
  you fail         12 (tp)       3 (fn)    = 15
  you pass          3 (fp)      22 (tn)    = 25
                  = 15         = 25         40

Why accuracy is not the answer

You agreed on 34 of 40, so the judge is 85% accurate. That sounds usable.

Now take a judge that does nothing but say "pass" to everything. On this same set it scores 62.5%. On a realistic production suite where only 5% of answers are bad, that same do-nothing judge scores 95%.

Accuracy rewards guessing the common answer. It cannot separate a judge that works from one that has noticed most things are fine. That is the whole reason this page exists.

Kappa: subtract the luck

You flagged 37.5% of examples. The judge flagged 37.5%. If both of you were flipping weighted coins at those rates and never reading the answers, you would still agree 53.1% of the time.

So the question is not how often you agreed, but how much of the room above luck you closed.

  room above chance:   100%  -  53.1%   =  46.9%
  you actually closed:  85%  -  53.1%   =  31.9%

  kappa = 31.9 / 46.9 = 0.68

In one line: kappa = (agreement observed − agreement by luck) / (1 − agreement by luck). One is perfect, zero is no better than coin-flipping. The do-nothing judge above scores exactly 0.00, which is the point.

How we read it, and where the thresholds came from

These bands are a convention, not a law. They descend from Landis and Koch (1977), who proposed them for observer agreement in medical data and said plainly that the divisions were arbitrary. We kept the shape because it is the one most readers already know, and collapsed the lower three into "not usable" because for our purpose the difference between fair and slight does not change what you do. If your domain has its own convention, use that instead.

The two numbers you act on

Kappa says whether the judge is worth anything overall. These say how it fails, which is what changes a rubric.

They trade off. Push a judge to catch more and it cries wolf more. A judge with a respectable kappa can still be unusable in the direction you happen to care about, which is why the report gives you all three rather than a single score.

The report also carries PPV (when it flags something, how often it is right) and NPV (when it passes something, how often it is right). Those are the ones to read if you are deciding how much to trust an individual verdict rather than the judge as a whole.

The interval, and why Wilson

12/15 is not "80%". It is 80%, measured on fifteen examples, and those are different claims. So every rate is printed with the range the true value probably sits in.

The textbook interval, the one most people are taught, would say:

  0.8 ± 1.96 × √(0.8 × 0.2 / 15)   =   [59.8%, 100.2%]

100.2%. It runs off the end of the scale, because that formula assumes a bell curve and a proportion near a boundary is not one. At 15/15 it collapses to [100%, 100%], claiming certainty from fifteen examples. At 1/15 it goes negative.

The Wilson interval does not do that. It stays inside 0 to 1, goes asymmetric near the edges, and holds its coverage down to single-digit counts. Your actual catch-rate interval is [54.8%, 93.0%].

That width is the honest content of the measurement. The judge's true catch rate could be 55%, and forty labels cannot rule it out.

Matthews correlation, in one line

0.68 here. A single correlation between the two raters' verdicts, from −1 to +1, which unlike accuracy degrades honestly when one class dominates. It is there for when someone wants one number, and it is a better one number than accuracy.

Where you disagreed

Six disagreements is a count. It does not tell you what to fix, so the taxonomy groups them by which rubric criterion the judge cited and which slice of your traffic they fell in, then refuses to report a slice whose interval does not clear the base rate.

In this run topic=returns disagreed 33% of the time against a 15% base rate, a 2.2x lift, on nine examples, with an interval of [12.1%, 64.6%]. The bottom of that interval sits below the base rate, so it is not reported. That guard is the most important thing in the codebase and it has its own page: disagreement taxonomy.

Did the rubric change help?

The taxonomy tells you where the judge disagrees. You act on it by rewriting the rubric — and then you have two calibrations and one question: did that help, or did it just move the noise around? langchef calibrate diff answers it. It re-scores the revised rubric against the same labels and reports the change in kappa and in both rates, each with an interval.

The worked example below uses a larger labelled set — sixty rather than the forty above — for a reason worth knowing before you run this. On forty labels, a revision that repairs two false alarms lands at +0.10, interval [0.00, 0.25]: inconclusive. Forty labels are enough to measure a judge and not enough to resolve a small change to one.

$ langchef calibrate diff --rubric answer-quality-v2
rubric delta on 60 labelled example(s) from run base
  answer-quality@290335165c70  ->  answer-quality-v2@8ad25d2ca381
  kappa  +0.57 -> +0.84   +0.27 [+0.11, +0.45]   IMPROVED
  TPR    80.0% -> 80.0%   +0.0% [+0.0%, +0.0%]   p=1.0000  inconclusive
         (nothing under 25.1% was in reach on these labels)
  TNR    80.0% -> 100.0%   +20.0% [+7.5%, +32.5%]   p=0.0156  improved
  moved  0 miss(es) fixed, 0 introduced; 8 false alarm(s) fixed, 0 introduced

In plain words: dropping the criterion the judge was reading as word-containment stopped all eight false alarms and cost nothing in catch rate. The moved line is the actionable one — a revision that fixes four false alarms by introducing four misses is not the same change as this, and kappa alone cannot tell them apart.

This is the same defect as the minimum detectable effect computed with an unpaired formula on paired data, which reported 15.6% where the truth was 6.0%. It does not crash and it does not look wrong. It quietly turns real improvements into shrugs, which is how a rubric-iteration loop stops being worth running.

The two rates get an exact test, not a bootstrap

Kappa is not an average of anything, so its interval is a bootstrap that resamples examples and carries the human label and both verdicts on every draw. The catch rate and the false-alarm rate are simpler than that. Because the human labels do not move, the human-fail examples are literally the same examples under both rubrics: the denominator is fixed and only the numerator can move. That is exactly McNemar's setting, so only the examples where the two rubrics disagree with each other carry information — the same arithmetic, and the same code, that compare uses on two arms.

Two more things the output commits to. Kappa is the headline and the two rates are its halves, so their p-values are Holm-corrected across that family of two and a direction is named only when the corrected p clears alpha and the whole interval agrees with it — never one without the other. And a delta needs the same judge model at both ends: move the model between the two and the command exits 5 rather than reporting a rubric change that is partly a model change.

The revised rubric does not need approving first, and deliberately so. Gate one stops a rubric nobody has read from scoring a suite for real; this command produces the evidence that reading is supposed to rest on. The output records that the candidate is unapproved, so nothing downstream can mistake it for a signed-off instrument.

What the rubric has to do

Each ### heading in your rubric is one criterion, and the judge must name the one it failed on. That is what makes the taxonomy possible at all: without an attributed failure there is nothing to group by, and a regression in a retrieval app cannot be told apart from a regression in the generator.

It is also why editing a rubric revokes its approval. The criteria are the axis the taxonomy is reported along, so renaming one silently changes what every past number meant.

What we deliberately do not do

• No weighted kappa. Verdicts here are pass or fail, so there is no notion of a near-miss to weight. If task classes with ordered labels arrive, this is where it changes.
• No multi-class kappa yet. The variance formula is written for a 2×2 table. Classification datasets with more than two labels need an N×N version, tracked as part of the bring-your-own-dataset work.
• A delta compares two rubrics, never two label sets. calibrate diff is paired on the labelled examples the two calibrations share, and refuses when they share none: two calibrations on different examples differ by rubric and by example set, and no interval can separate those two causes.
• No inter-annotator agreement. We compare one judge against one set of human labels. Multiple human labellers disagreeing with each other is a real and harder problem, and Krippendorff's alpha is the usual tool. Out of scope for now.
• The stratified label weights are recorded and are not valid weights. This is the sharpest limitation on this page, and it is stronger than "not yet applied". The label plan sorts each stratum by the judge's confidence and takes the lowest, so a row's inclusion probability is 0 or 1 rather than n/N. Post-stratification cannot repair that, because selection inside a stratum tracks the very thing being estimated. A seeded coverage simulation put a nominal 95% interval at 43% actual coverage for the catch rate. So read every rate on this page as describing the labelled set, not as an estimate of your whole suite. The underlying question, whether one label budget buys estimation or diagnosis, was settled on 2 September 2026: it buys diagnosis. These rates are screening statistics measured on the cases the judge was least sure about, which biases agreement downward and makes them conservative for the question you are actually asking, which is whether the judge can be trusted at all. Estimating a population rate is a separate opt-in mode that refuses when the label budget cannot support the width you asked for.
• scikit-learn is a test dependency, never a runtime one. Every statistic is hand-rolled and checked against an independent implementation. If the product and its check came from the same library, the check would only prove the library agrees with itself.

Further reading

Ordered by how useful they are if you are actually changing this code.

• Brown, Cai and DasGupta (2001), "Interval Estimation for a Binomial Proportion", Statistical Science 16(2), 101–133. The paper that settles why not to use the textbook interval: it shows the Wald interval's coverage is chaotic and that the usual reassurances about when it is safe are "misleading and defective". It recommends Wilson for small n, which is what we use. Start at §1.1 and Figure 1; the picture makes the argument on its own. Open PDF: www-stat.wharton.upenn.edu/~lbrown/Papers/2001a Interval estimation for a binomial proportion.pdf
• Cohen (1960), "A Coefficient of Agreement for Nominal Scales", Educational and Psychological Measurement 20(1), 37–46. The original kappa. Short, and the motivating argument in the opening pages is the same one this page makes about accuracy.
• Fleiss, Cohen and Everitt (1969), "Large sample standard errors of kappa and weighted kappa", Psychological Bulletin 72, 323–327. The asymptotic variance kappa_interval() implements. Read this one if you touch that function; the index gymnastics in the published form is where implementations go wrong, which is why our test checks it against a bootstrap. doi:10.1037/h0028106
• Landis and Koch (1977), "The Measurement of Observer Agreement for Categorical Data", Biometrics 33(1), 159–174. Where the slight / fair / moderate / substantial / almost perfect bands come from. Worth reading precisely because they present the divisions as arbitrary, which is not how they are usually cited.
• Chicco and Jurman (2020), "The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation", BMC Genomics 21:6. Open access and the most readable thing on this list. Worked examples of accuracy and F1 flattering a bad classifier, which is the same failure this page opens with.
• Zheng et al. (2023), "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", arXiv:2306.05685. Not about kappa, but about the instrument. §3.1 catalogues position bias, verbosity bias and self-enhancement bias in model judges. Read it before you trust a judge you have not calibrated, and before you let a model grade its own family's output.

Citations verified against the publishers' records rather than quoted from memory. Where only a section could be confirmed, a section is what is cited.