Reference

Comparing two arms

How LangChef decides whether a change helped, hurt, or cannot be told apart, and why "cannot be told apart" is a result rather than a shrug.

The worked example

From the dogfood: the baseline against an arm with documents deliberately dropped from the retrieval index. Both arms answered the same 90 questions.

  baseline 83.3%   variant 63.3%
  difference -20.0% [-27.8%, -12.2%]  p=0.0000
  REGRESSION

  of 90 goldens, 18 changed verdict: 18 broke, 0 fixed

The planted effect was −20 points. The measurement recovered it to the decimal.

Why paired, and not two samples

Both arms answer the same questions, with the same example ids. That is not a convenience, it is the whole reason a set of 90 can say anything.

Think about what actually carries information. If 72 goldens pass under both arms, they tell you nothing about the difference between them: they are the same question, answered acceptably twice. The evidence lives entirely in the 18 that changed verdict.

                variant pass   variant fail
  baseline pass       72             18      <- broke
  baseline fail        0              0      <- fixed

The test asks one question about those 18: given that a verdict flipped, was it equally likely to flip either way? Eighteen broke and none were fixed. If the two arms were really equivalent, that is a coin landing heads eighteen times.

Treating the arms as two independent samples instead throws the pairing away, inflates the variance, and real regressions come back as "not significant". That failure is quiet and it is common.

Computed in src/langchef/core/compare.py

discordance() the paired 2×2; only the off-diagonal moves the estimate

mcnemar_p() exact binomial on the discordant pairs

compare() the verdict, the interval, and the detection limit

by_criterion() the same test inside each criterion, corrected across them

holm() step-down adjusted p-values for the family of criteria

Exact, not the chi-square approximation

The classical McNemar statistic uses a chi-square approximation. We use the exact binomial test instead, because the discordant counts that decide real cases are small, and the approximation is least reliable exactly when the decision is closest. With 18 flips the two agree; with 4 they do not, and 4 is a number you will see.

The verdict comes from the interval, not the p-value

Three outcomes, and the middle one is the one people misread.

Verdict	Condition	Means
REGRESSION	whole interval below zero	worse, and the evidence supports acting
IMPROVEMENT	whole interval above zero	better, same standard
INCONCLUSIVE	interval spans zero	this set cannot tell them apart. Not "no difference"

A direction only counts when the entire range agrees with it. The p-value is reported because people ask for it, but it does not decide anything here: a p-value answers "how surprising is this if nothing changed", which is not the question you came with.

The interval itself is a percentile bootstrap over the pairs, resampled together so the pairing survives. It is seeded, because the contract calls compare deterministic and a comparison that moves between identical runs is not one.

The smallest change you could have seen

This is the most useful number in the tool, and the one no dashboard gives you.

  difference +0.0% [+0.0%, +0.0%]  p=1.0000
  INCONCLUSIVE
  (smallest effect this run could have seen: 6.0%)

That run came from an arm with a real, deliberately planted −3.3 point regression. The judge saw nothing. The honest report is not "no regression found", it is "nothing we could have seen", and the second half of that sentence is the actionable part: ninety goldens could never have resolved three points.

It is computed from the discordant rate, not the pass rate, because under a paired test that is what carries the information. Two hundred goldens where four flip carry far less than two hundred where forty do, and a formula built on the pass rate cannot see that difference.

When nothing flipped at all, the rate is not taken as zero. Zero discordant pairs is not evidence that any effect was detectable; it is an unknown rate that this many goldens bound from above, so the upper end of its interval is used. That is why a completely quiet run still reports a finite limit rather than claiming infinite sensitivity.

Rule of thumb: halving the effect you want to detect needs roughly four times the goldens.

Which criterion moved

"Quality fell 11 points" is a fact. "The generator got more cautious and retrieval held" is an answer. The judge already names the criterion it cited on every failure, so the comparison can be attributed rather than left as one number over two systems.

  attribution over 2 criterion(s), Holm-corrected — not 2 separate findings:
    Directness     -16.7% [-24.4%, -8.9%]  p=0.0001  MOVED WORSE
    Correctness    +5.6% [+1.1%, +11.1%]  p=0.0625  inconclusive
                   (nothing under 12.0% was in reach for this criterion)

That is the dogfood's eager-hedging arm: the app was made to decline more often, and nothing else was touched. The overall drop is 11 points; the breakdown puts 17 of them on Directness and finds no movement in Correctness it can stand behind.

The pairing is inside a criterion, never across two

Each criterion gets the same treatment as the whole suite: the same example under both arms, the same exact McNemar over the pairs that flipped on that criterion, the same seeded bootstrap. Nothing switches to a two-sample test on a smaller slice.

This matters most for the case the headline cannot see at all. An answer that failed Correctness under the baseline and Groundedness under the variant is a fail in both arms — zero discordant pairs, nothing to report — while the criteria show a Correctness fix and a Groundedness break. Something moved, and only the breakdown says what.

Because the judge cites exactly one criterion per failure, the per-criterion differences add up to the overall difference. When a judge fails something without naming a criterion, the leftover is reported as unattributed rather than quietly dropped.

Five criteria means five chances to be wrong

With five criteria and a 5% threshold, one of them crossing by chance is ordinary — about a one-in-four run. Printing five uncorrected verdicts would manufacture a finding roughly every fourth time you ran the tool, and every one of them sends somebody to rewrite a component that was fine.

So the p-values are Holm-corrected across the criteria examined. Sort them, hold the smallest to α/k, the next to α/(k−1), and so on; the family-wise error rate stays at 5% no matter how the criteria are correlated, and they are correlated here — one cited criterion per failure makes them push against each other.

Holm rather than Bonferroni because it is uniformly more powerful at the same guarantee. Holm rather than a false-discovery-rate procedure because these criteria are the two or three halves of one system: the question is which one broke, so a false positive costs a person a day in the wrong file, and that is the error rate worth controlling.

Different words, on purpose

A criterion is never called a regression or an improvement. It is moved_worse, moved_better or inconclusive. Those are attributions of one comparison, and the vocabulary says so in the payload rather than in a footnote a reader drops on the way to the number.

A direction is only named when both the corrected p-value clears the threshold and the interval sits entirely on one side of zero. Either one alone can mislead: five flips the same way is p = 0.0625 and cannot be rejected, but its bootstrap interval still clears zero. That is the Correctness line above — the interval alone would have called it an improvement.

Each criterion carries its own detection limit

This is the part that is easy to get wrong. The overall detection limit was computed for the overall comparison at the overall threshold; quoting it beside a per-criterion line it cannot support is the failure mode here.

So every criterion reports its own, computed at α/k — the strictest rung of the ladder, and the one a lone signal actually faces. It is not automatically wider than the overall limit: a criterion that few examples flipped on is estimated more precisely, not less, and the multiplicity price pushes the other way. Both effects are real, so the number is computed rather than assumed, per criterion, every time.

The attribution is only as good as the judge doing the citing

A judge names one criterion per failing example, so an example that failed Correctness tells you nothing about whether it would also have failed Groundedness. A criterion is credited with a failure only when it was cited. That is a real limit, and it is the second reason this is reported as attribution rather than as a set of independent per-criterion measurements.

Did quality hold

Swapping in a cheaper or smaller model is a different question and reading it the same way will mislead you. You are not hunting an improvement; you are checking the drop stays inside a tolerance.

Set the margin before the run. Then read the bottom of the interval against it, not the middle:

  difference -1.2% [-4.1%, +1.7%]
  INCONCLUSIVE

The middle looks fine, barely down. But the range reaches −4.1%, past a three-point tolerance. This run has not shown quality held. It has shown that a drop big enough to matter is still consistent with what was measured.

Three outcomes: held when the whole interval clears the margin, failed when it is entirely past it, unresolved when it straddles. Unresolved is not permission to ship.

Why the margin must be pre-registered

A margin chosen after seeing the interval is not a tolerance, it is a rationalisation. This is why the design is written to evals/experiments/, hashed, and approved by a person before any traffic, and why editing it afterwards revokes the approval. The discipline is the feature; the arithmetic is easy.

What we deliberately do not do

No interim looks, no sequential testing. The stopping rule in every design is "score them all, then read out once". Peeking at a running experiment and stopping when it looks good inflates the false-positive rate badly. Always-valid sequential bounds are a real answer to this and are not built yet.
No correction across arms. The per-criterion breakdown is corrected across criteria, because those k tests are run together and we know k. Sweeping ten variants and reporting the best is the other multiplicity problem, and this tool does not see it: it compares two arms at a time and cannot know how many comparisons you already ran. That correction is yours to make, and it is a known way to find an effect that is not there.
No simultaneous intervals. Each criterion's interval is at the nominal level, so the k of them do not jointly cover at 95%. The correction is applied to the tests, which is what gates the attribution; the intervals are printed for magnitude. Reading them as k separate verdicts is the mistake the wording is chosen to prevent.
Retrieval is compared on continuous scores, not thresholded. recall@k, MRR and nDCG go through a paired comparison of their own: Wilcoxon signed-rank with a bootstrap interval over the per-example differences, because those scores are bounded, pile up at the ends, and are nowhere near normal at the sizes anybody labels. Reducing them to pass or fail was considered and rejected: recall 0.42 and 0.71 are both "fail" against a threshold of 0.8, and the gap between them is the entire finding. See DECISIONS #12.
Classification needs no reduction either. Its per-example outcome is predicted == ideal, which is binary natively, so the paired test above applies unchanged and nothing is lost. Multi-class kappa is a non-goal: it measures agreement between two raters, and a hard label has no second rater.
No graded relevance. nDCG here is over binary relevance. A document is relevant or it is not. Graded judgements are a real thing and this does not do them.
No bandits over judge scores. Allocating traffic by a judge's own score rewards whatever the judge is biased toward, including verbosity and position. If it is ever done here, quality non-inferiority is established first and the bandit runs on cost and latency only.