Reference

Disagreement taxonomy

Six disagreements is a count. This is the part that turns it into something you can act on, and the part that refuses to when the evidence will not carry it.

The worked example

The same 40 labelled examples as the agreement page: your judge and you disagreed six times, three misses and three false alarms, a base rate of 15%. Knowing that number tells you nothing about what to change.

Two shapes, and they do not cost the same

Every disagreement is one of two things, named from the judge's point of view.

Shape	What happened	What it costs
false alarm	judge said fail, you said pass	Trust. Every one is a person going to look at something that turned out to be fine. Enough of them and nobody reads the reports.
miss	judge said pass, you said fail	The whole suite. These are the regressions that ship. One miss can cost more than fifty false alarms.

They are reported separately because the fix differs. False alarms usually mean a criterion is worded too strictly. Misses usually mean the rubric never mentioned the thing that went wrong.

Which criterion is at fault

Each ### heading in your rubric is one criterion, and the judge names the one it failed on. Group the disagreements by that name and the six become a diagnosis:

  Directness     2/3 =  66.7%   [20.8%, 93.9%]   2 false alarms
  Groundedness   1/8 =  12.5%   [ 2.2%, 47.1%]   1 false alarm
  Correctness    0/4 =   0.0%   [ 0.0%, 49.0%]

Read the denominators: they are the examples where the judge cited that criterion, not your suite. So the middle column says when this judge invokes Directness, it is wrong two times in three. That is a rubric edit, and a specific one. Correctness is behaving.

Now read the interval. Two of three rests on three examples, and the true rate is somewhere between 21% and 94%. The direction is worth acting on because the fix is cheap; the number is not worth quoting to anybody.

This axis cannot see misses

A miss is the judge saying pass. A judge that passed an example named no failing criterion, so there is nothing to group by. Every disagreement in the table above is a false alarm, and that is structural rather than a property of this data.

Which is awkward, because misses are the expensive kind. Attributing them would need the judge to report which criteria it considered and cleared, not only the one it failed. The provider shim already computes coverage of that shape for a different purpose, so the path exists and is not wired up. Until it is, read the criterion table as an account of why your judge cries wolf, and nothing more.

Which part of your traffic

The other axis is whatever slice metadata came with your examples. Same six disagreements, cut by topic:

  returns    3/9  = 33.3%   [12.1%, 64.6%]
  accounts   2/16 = 12.5%   [ 3.5%, 36.0%]
  shipping   1/15 =  6.7%   [ 1.2%, 29.8%]

  base rate across all 40: 15.0%

There it is. Returns disagrees 33% of the time against a 15% base rate. A 2.2x lift. That is the line that sends somebody off to investigate for a day.

Why the tool will not report that

Look at the interval on the returns row: [12.1%, 64.6%]. The bottom of it, 12.1%, sits below the 15% base rate. The evidence is consistent with returns being no worse than anything else.

So concentrations() marks it separated: false and the memo leaves it out. The rule is one line:

  separated = worst.interval.lo > base_rate

The most important guard in this codebase

Every slice report has a worst row. That is arithmetic, not a finding. Sort three topics by disagreement rate and one of them is 2.2x the average whatever the data says, because something has to be first.

A tool that names the worst-looking slice every time is a random number generator with good manners. It will be right occasionally, which is worse than being wrong reliably, because the occasional hit is what convinces people to keep trusting it.

There is a second guard beside it. A slice with fewer than min_bucket examples, five by default, is not ranked at all. One disagreement out of two is a 50% rate and a 3.3x lift, and it is nothing.

When a concentration does separate, it is reported with its interval attached, so you can see whether "twice as bad on long answers" rests on four examples or four hundred.

What to do with each finding

What you see	What it means	What to do
One criterion holds most of the false alarms	That criterion is worded too strictly	Edit its wording, re-approve, re-run. The rubric hash changes, so this is tracked
Misses dominate and no criterion explains them	The rubric never mentioned the failure mode	Read the missed examples and add a criterion
A separated concentration	One part of your traffic really is harder	Label more there, or split the suite and calibrate separately
A concentration that did not separate	You have not learned anything yet	Nothing. This is the finding you were about to waste a day on

Computed in src/langchef/core/taxonomy.py

Judgement.kind classifies each row as miss, false_alarm, or agreement

by_criterion() groups by the rubric heading the judge cited

by_slice() groups by one metadata dimension, worst rate first

concentrations() ranks dimensions and applies the separation test

summarise() the whole thing as plain data for the memo

What we deliberately do not do

No correction for the number of slices examined. The separation test is a per-slice filter, not a family-wise one. Cut your data forty ways and it is still doing forty independent checks. The honest mitigation today is that slice dimensions come from metadata you already had, rather than being searched for. If automatic slice discovery is ever added, a correction has to arrive with it.
No interaction terms. Returns-and-long-answer might be far worse than either alone. Finding that reliably needs more labels than anyone is going to give us.
No causal claim. A separated concentration says disagreement clusters there. It does not say the topic caused it. Returns questions may simply be longer, and length may be the real driver.
No automatic rubric editing. The taxonomy points at a criterion. A person writes the new wording and approves it. A tool that rewrites the definition of "good" in response to its own error pattern is optimising for its own agreement.
Misses are not attributed, as above. This is the largest gap on the page.