Reference

Judging and pins

The judge is a measuring instrument, not a metric. This page is how one is defined, how it is pinned so two runs can be compared, and how it is stopped from spending money you did not agree to.

The worked example

One example, scored against a two-criterion rubric by the judge that ships in the box.

  question   Do you ship to Canada?
  answer     Yes, standard shipping to Canada takes 5-7 business days.
  context    We ship to Canada. Standard delivery is 5-7 business days.

  verdict     fail
  confidence  0.6
  criterion   Groundedness
  rationale   40% of the answer's content words appear in the retrieved context

Three things to notice. It named which criterion failed, which is what makes the taxonomy possible. It gave a confidence, which is what drives escalation and label planning. And it gave a rationale you can argue with, which is what makes a wrong verdict findable.

The rubric is the definition of good

A rubric is Markdown. Each ### heading is one criterion, and the judge must name the one it failed on.

### Groundedness
Every claim traceable to the context.

### Correctness
Answers the question asked.

  criteria parsed:  ('Groundedness', 'Correctness')
  rubric ref:       support@4f2f2ee68288

That ref is a name plus a content hash, and it is the pin for the rubric. Change three words in one criterion and it moves:

  support@4f2f2ee68288   ->   support@d7f133b168ab

Which is the point. Editing a rubric changes what "good" means, so it revokes its own approval and invalidates every cached verdict produced under the old wording. You cannot quietly loosen the definition and compare against last week.

A judge can only cite criteria you wrote

In the worked example the rubric has Groundedness and Correctness and no criterion about hedging. Feed it "I'm not able to say for certain whether we ship to Canada" and it returns fail against Correctness, because that is the closest thing in your rubric to what went wrong.

The verdict is right and the attribution is misleading, and no amount of calibration will surface it, because you and the judge agree on the verdict. If your taxonomy keeps blaming one criterion for unrelated failures, the rubric is missing a criterion.

Pins: what produced these numbers

Every run records the instrument that produced it.

Field	Why it is in the pin
`rubric`	The name and content hash. A different definition of good is a different measurement
`provider`	Which backend produced the verdicts
`cheap_model`	The model that scored everything
`strong_model`	The model that re-scored the unsure ones, if any

Compare two runs whose pins differ and the tool exits 5 and refuses, naming what moved:

  pin moved — cheap_model: 'gpt-4o-mini' -> 'claude-haiku-4-5'

This is the difference between measuring a change in your app and measuring a change in your ruler. Both look like a moving number. Only one of them is a finding.

The cache, and what is in the key

Verdicts are content-addressed. The key covers everything that changes what a correct verdict is:

  example_id, question, answer, context, expected, rubric ref, model, tier

Slices are deliberately not in the key. They are metadata for grouping, so adding a topic tag to your examples would otherwise invalidate every cached verdict and re-buy the whole run for nothing.

Tier is in the key, so the same example judged cheaply and judged strongly are two entries rather than one overwriting the other:

  cheap tier   fd3da9b9f7259e30...
  strong tier  47eb382ec2f5e922...

The practical effect is that re-running an arm after a change is nearly free, and only genuinely new work costs anything. It is also why the integrity gates exist: with a warm cache, re-running an arm until it reads out better costs nothing, which is exactly why readout refuses when more than one run matches.

Two tiers

A cheap model scores every example. Anything it scored below 0.6 is re-scored by a strong model, and only those.

The confidence in the worked example is exactly 0.6, so it would not escalate. The threshold is a floor, not a ceiling.

This was built in from the start rather than added later, for a specific reason: the model is part of the cache key. Retrofitting a second tier would have invalidated every cached verdict in every workspace on the day it shipped.

Budgets, and stopping before the bill

A run can carry a ceiling on provider calls, not on examples. A cached example is free and does not count against it.

When the ceiling is hit the run stops where it is, writes what was left unscored to runs/<id>/undone.json, and exits 4. It does not silently produce a partial result that looks whole.

The reasoning is that a partial result whose shape you know is worth more than a bill you did not agree to. An agent handed exit 4 knows to stop and ask rather than to retry.

The judge in the box

The default provider is not a model. It is deterministic token containment: what fraction of the answer's content words appear in the retrieved context, and does the expected string appear. It needs no API key, no network, and no money.

It is genuinely weak, and that is deliberate. It exists so the whole pipeline can be run, tested and dogfooded end to end before anyone spends anything, and so the test suite never depends on a provider being up. Its version string is part of the cache key, so changing its checks without bumping that version would leave every warm cache serving verdicts from the old logic.

src/langchef/judge/rubric.py — parse, criteria, digest, ref

src/langchef/judge/providers.py — the single seam a backend plugs into

src/langchef/judge/cache.py — judgement_key(), what is and is not in it

src/langchef/judge/runner.py — Pin, escalation, the budget ceiling

What we deliberately do not do

No position or verbosity mitigation. Model judges favour the first answer shown and longer answers regardless of content. We do not swap positions or normalise length. The defence here is different: calibration measures how far your judge disagrees with you, so a judge with a verbosity bias shows up as a bad kappa rather than as a silent skew. That is a weaker defence than fixing the bias, and it is honest about being one.
No model grading its own family's output. Self-enhancement bias is real. If the judge and the system under test share a model family, calibrate before believing anything.
No scores, only verdicts. Pass or fail, with a confidence. A judge returning 7.4 out of 10 invites arithmetic that its own precision cannot support.
No prompt tuning loop. Nothing here rewrites your rubric to improve agreement. A tool that edits the definition of good in response to its own error rate is optimising for its own agreement, and the number stops meaning anything.
No retries on a disagreeing verdict. Asking again until it agrees is not measurement.