What every number means

Plain English first, the textbook name second. You do not need any of this to use the tool — it is here for when a number looks wrong and you want to know what it is telling you.

Can this judge be trusted?

calibration for base on 40 labelled example(s)
  kappa      0.68  0.44..0.92
  TPR        80.0%  (12/15)
  FPR        12.0%
  disagreed  6 ({'false_alarm': 3, 'miss': 3})

Agreement — the kappa line

How much the judge agrees with you, after subtracting the agreement you would get by luck. Runs from 0 (no better than guessing) to 1 (perfect).

Why not just "it agreed with me 90% of the time"? Because on a suite where 95% of answers are fine, a judge that says "fine" to everything agrees with you 95% of the time and is worthless. This number sees through that; a raw percentage does not.

Textbook name: Cohen's κ.

Catch rate — the TPR line

Of the problems you found by hand, how many did the judge also flag? Here, 12 of the 15. The other three shipped past it.

The counts matter as much as the percentage: "80%" reads like a solid measurement until you see it rests on fifteen examples. That is why the tool always prints (12/15) next to it.

Textbook name: true positive rate, or recall.

False alarm rate — the FPR line

Of the answers that were actually fine, how many did the judge flag anyway? Every one of these is somebody investigating a non-problem. A judge with a high false alarm rate gets ignored within about two weeks, which is worse than having no judge.

Textbook name: false positive rate.

The range after each number

0.44..0.92 is not decoration. It is the honest width of what forty labels can tell you: the true agreement is probably somewhere in there. A wide range means "label more examples", not "the judge is inconsistent".

Textbook name: a 95% confidence interval — Wilson for rates, asymptotic for agreement.

Where it disagreed

The report also groups the disagreements: which rubric criterion the judge cited when it was wrong, and whether any slice of your traffic is worse than the rest. A line like topic=returns disagrees 33% of the time against a 15% base rate is the actionable form of "the judge is unreliable".

Did the change help?

base -> variant on 90 shared golden(s)
  baseline 83.3%   variant 63.3%
  difference -20.0% [-27.8%, -12.2%]  p=0.0000
  REGRESSION

The verdict

The difference, and its range

-20.0% [-27.8%, -12.2%] — the pass rate dropped twenty points, and the true drop is probably between twelve and twenty-eight. The verdict is decided by the range, not the p-value: a direction only counts when the whole range agrees with it.

How many examples actually changed

Both versions answered the same questions, so what matters is how many flipped. If 200 examples pass under both versions and 3 flip, the entire evidence is in those 3. The tool reports that count, and everything is computed from it.

Textbook name: discordant pairs, tested with an exact McNemar test. The pairing is why this is not a two-sample proportion test — treating the arms as independent throws away the fact that they answered identical questions, and real regressions then come back "not significant".

The smallest change this run could have caught

  INCONCLUSIVE
  (smallest effect this run could have seen: 6.0%)

The single most useful number in the tool. It converts a shrug into a plan. Your run could not have detected anything under six points — so if you were hoping to catch a two-point regression, you need more examples, and no amount of re-reading this output will change that.

Roughly: to halve the smallest detectable change, you need about four times the examples.

Textbook name: minimum detectable effect, computed from the flip rate rather than the pass rate, because under a paired test that is what carries the information.

"We swapped in a cheaper model — did quality hold?"

This is a different question from "is it better", and reading the output the same way will mislead you. You are not hoping for an improvement. You are checking that the drop is small enough to accept in exchange for the cost or latency you bought.

Decide the tolerance before the run. Say you will accept up to a three-point drop for a 70% cost reduction. Then read the bottom end of the range, not the middle:

  difference -1.2% [-4.1%, +1.7%]
  INCONCLUSIVE
  (smallest effect this run could have seen: 5.8%)

The middle looks fine — barely down. But the range reaches −4.1%, past the three points you were willing to lose. This run has not shown that quality held. It has shown that a drop big enough to matter is still consistent with what you measured.

Two honest ways forward: add examples until the range tightens inside your tolerance, or accept the risk deliberately and write down that you did. What you should not do is read "inconclusive" as "no difference" and ship it as proven.

You do not have to do it by hand. Pass the tolerance and the tool applies it:

langchef compare --variant new-model --tolerance 0.03

  difference -1.2% [-4.1%, +1.7%]  p=0.4127
  INCONCLUSIVE
  against a 3.0% tolerance: QUALITY UNRESOLVED
    unresolved is not held. This run could not resolve 3.0%;
    it needed to see 6.0% or larger.

The three answers, in plain words

When the margin was decided is the whole mechanism

A tolerance passed on the command line and one carried in a pre-registration produce the same three words. They are not the same evidence. A margin set before the run constrains the person who set it; one typed after seeing the interval constrains nobody, and can be retyped until the answer is agreeable.

So the output says which it was, every time:

    margin came from the command line, not a pre-registration,
    so it constrains nobody: it could have been chosen after
    seeing the interval above.

And if the run belongs to a pre-registered experiment, a --tolerance that disagrees with the registered margin is refused at exit 2 rather than preferred. Otherwise the one command that sits outside the gate would be the way around it.

Textbook name: a non-inferiority test. One detail worth knowing: the interval is two-sided at your confidence level, and this test reads one bound of it, so the effective one-sided level is higher than the number quoted. That is the conservative direction, and the payload reports both rather than letting you assume.

Which stage broke?

When a retrieval app regresses, the number you get back is one number over two systems. The rubric is how you pull them apart: each criterion is a different failure mode, and the judge names the one it failed on for every example.

compare reports that split for you. Under the overall verdict it prints one line per criterion, so the answer to "which half broke" arrives with the answer to "did it break".

  difference -11.1% [-17.8%, -5.6%]  p=0.0020
  REGRESSION
  attribution over 2 criterion(s), Holm-corrected — not 2 separate findings:
    Directness     -16.7% [-24.4%, -8.9%]  p=0.0001  MOVED WORSE
    Correctness    +5.6% [+1.1%, +11.1%]  p=0.0625  inconclusive
                   (nothing under 12.0% was in reach for this criterion)

Read that as one finding with a location, not as two results. The generator got more cautious; retrieval did not move in a way this run could resolve. The second line is the one that saves a day of looking in the wrong place.

Every scored example records its criterion in runs/<id>/scores.parquet, so the calibration report groups by criterion and by slice as well — the same axis, one run at a time instead of across a pair.

Was this even a fair comparison?

Every run records the rubric hash, the provider and the models that produced it. Together those are the pin — the exact ruler used.

$ langchef compare --variant later-run
langchef: pin mismatch — rubric 'answer-quality@290335165c70'
-> 'answer-quality@b12aabd7ee8e' — these are two measurements, not a
comparison. Re-run the older arm under the current pin.

You edited the rubric between the runs, so the numbers were produced by different instruments. The tool stops rather than drawing the chart. Re-score the older version under the current rubric and compare again — the cache makes that cheap.

If your task has a hard target

Retrieval, classification and reranking come with the right answer already known, so the calibration half of this page does not apply to you. There is no rubric, no judge and no kappa, because nothing is being judged. Skip every number above that describes agreement.

What still applies, and is most of why this tool exists:

The per-task metrics themselves, recall@k, MRR, nDCG for retrieval and accuracy, precision, recall and F1 for classification, are computed from your targets rather than judged. They are designed and not yet shipped: see #14.

When the tool refuses

Refusals are exit codes, so a script or an agent cannot talk its way past them.

Codes 3 and 4 are reserved for abstention and spend caps and are not emitted yet. The full table, generated from the binary, is on the commands page.

The memo

langchef memo render assembles all of the above into one page, in a fixed order: whether the judge can be trusted, then the result, then what the run could not rule out. That order is deliberate. A confident result from an unchecked judge is worse than no result, so the trust question is never an appendix.