Start here

Your first evaluation

A worked example, start to finish. About forty-five minutes the first time, most of it spent labelling. Ten minutes per change after that.

The scenario

Your support assistant answers customer questions from your help centre, retrieval-augmented over your docs. Your provider is retiring the model you shipped on, so you have to move to the replacement. Does answer quality hold?

The same loop covers every other change in the stack — a new embedding model, different chunking, a reranker, a fine-tuned small model swapped in to cut the bill. Only the arm you compare against changes.

Check this walkthrough is for you

This page follows a free-text task: the assistant writes prose, there is no single correct answer, and so a judge has to read it. Steps 2, 3, 5 and 6 exist entirely to check that judge before you trust it.

If your task has a hard target, meaning you already know which document should have been retrieved, which label is correct, or which ordering is right, then none of those four steps apply to you. There is nothing to calibrate, because comparing against a known answer is arithmetic rather than judgement. Your version of this walkthrough is steps 1, 7, 8 and 9: collect examples, have the experiment designed, run it, read it out. That path is designed and not yet shipped (#14); the comparison, the detection limit and the pre-registration it depends on all exist today.

Which one am I?

Install — 2 minutes

uv is the only prerequisite. It fetches its own Python, so nothing is installed globally and no system package is touched.

curl -LsSf https://astral.sh/uv/install.sh | sh   # if you don't have it

git clone https://github.com/deepskandpal/LangChef.git langchef
cd langchef
uv sync
uv run langchef doctor

Want to see the whole thing work before pointing it at your own app? The repository ships a small search app with faults deliberately planted in it, so you can watch the tool find them:

uv run python -m dogfood.build
uv run pytest tests/test_dogfood.py -v

Words used on this page

Word	What it means here
Example	One question your app was asked, the answer it gave, and what it retrieved. Also called a golden. You need 50–100.
Rubric	What "a good answer" means, written down — roughly what you'd tell a new teammate. A Markdown file you review like code.
Judge	The thing that reads an answer and says pass or fail. Usually a model with your rubric in the prompt.
Label	Your own verdict on an example. The ground truth the judge is checked against.
Calibration	Comparing the judge's verdicts to your labels, to find out how far it can be trusted.
Arm	One version of your app — usually `baseline` against the thing you changed.

Step 1 — Collect examples · 10 minutes

An example is one question, the answer your app gave, and the context it retrieved. Pull 50–100 from your logs. Real questions, not invented ones — invented questions are always easier than the ones users actually ask, and a model migration tends to break on exactly the awkward ones.

cd your-project
langchef init

That creates an evals/ directory. Put your examples in evals/goldens/support.baseline.jsonl, one JSON object per line:

{"example_id": "q-001",
 "question": "How long do refunds take?",
 "answer": "Refunds are issued within ten working days of the parcel arriving back.",
 "context": ["Refunds are issued within ten working days of the parcel arriving back."],
 "expected": "ten working days",
 "slices": {"topic": "returns"}}

expected is the fact a correct answer has to contain — you know this because you picked the question. slices are optional tags; they are how the tool later tells you "this only breaks on billing questions".

Step 2 — Say what "good" means · 10 minutes

langchef init wrote a starter rubric at evals/rubrics/answer-quality.md. Open it and make it yours. It is Markdown, each ### heading is one criterion, and the judge has to name the criterion it failed on:

### Correctness

The answer states the fact the question asked for. An answer that is
merely adjacent to the right topic fails this criterion.

### Groundedness

Every claim is supported by the retrieved context. An answer that is
correct but not present in the context still fails: on a retrieval
system that is a lucky guess, and it will not stay lucky.

### Directness

The answer answers. Hedging into uselessness when the context contains
the answer fails here.

Notice what those three criteria are doing: one per failure mode, and they map onto the stages of your pipeline. Correctness is the generator. Groundedness is retrieval — an answer that is right but absent from the retrieved context means your index got lucky. Directness catches the model refusing when the answer was sitting right there, which is the classic way a smaller or newer model regresses.

Because the judge has to name the criterion it failed on, failures stay attributable instead of pooling into one number. When the new model scores worse, you can see whether it is answering badly or being handed bad context.

This is the highest-leverage twenty minutes in the whole process. A vague rubric produces a vague judge, and no amount of statistics downstream repairs that.

Step 3 — Sign it off · 10 seconds

Nothing will run until a person has read the rubric and approved it:

$ langchef judge run --arm baseline
langchef: refused — an approval gate is unmet — no rubric approved yet

$ langchef approve rubric
approved answer-quality@290335165c70
  - Correctness
  - Groundedness
  - Directness

That hash is the point. If you edit the rubric later, the hash changes, the approval lapses automatically, and the next run stops until you re-read it. This is what prevents trap three — the ruler moving without anyone noticing.

Step 4 — Score them · 30 seconds

$ langchef judge run --arm baseline
judge: n=90  provider_calls=90  pass=75  fail=15

Out of the box this uses a judge that needs no API key and no network — it checks whether the expected fact is present, whether the answer is grounded in the retrieved context, and whether it hedged. It is a real technique and a reasonable starting point, and it has a real weakness you are about to discover in step 6.

To use a real model as the judge instead, set a few lines in evals/config.toml:

[judge]
provider = "litellm"
cheap_model = "anthropic/claude-haiku-4-5"
strong_model = "anthropic/claude-sonnet-5"   # re-scores only the unsure cases

Any provider litellm speaks works here. One rule worth keeping: do not judge a model with itself. If you are migrating to a model, grading its answers with that same model measures its self-consistency, not its quality — which is another reason step 6 exists.

Step 5 — Label forty yourself · 20 minutes

This is the only manual work, and it is not optional. It is the only way to find out whether the judge is worth anything.

$ langchef label plan --budget 40
40 of 90 examples planned for labelling
  by stratum: {'fail': 15, 'pass': 25}
  -> evals/labels/answer-quality.todo.jsonl

It does not pick forty at random. It takes every example the judge flagged plus a sample of the ones it passed, because a random sample of a suite with a 17% failure rate spends most of your attention confirming things that were already fine.

Open that file, read each answer, and set "verdict" to "pass" or "fail" — your honest opinion, not what you think the judge said. Then:

langchef label import evals/labels/answer-quality.todo.jsonl

Step 6 — Find out if your judge is any good · 5 seconds

$ langchef calibrate report
calibration for base on 40 labelled example(s)
  kappa      0.68  0.44..0.92
  TPR        80.0%  (12/15)
  FPR        12.0%
  disagreed  6 ({'false_alarm': 3, 'miss': 3})

In plain words: this judge is usable but not great. It caught 12 of the 15 problems you found, cried wolf on 12% of the good answers, and agrees with you meaningfully more than chance would explain. Reading the output decodes each of those lines properly, including what to do when the number is bad.

If agreement had come back below 0.4, the correct move is to stop, fix the rubric, and try again — not to carry on and report pass rates from a judge that disagrees with you.

"Try again" has a command. Once you have changed the rubric, langchef calibrate diff re-scores the new one against the labels you already have and tells you whether agreement actually moved — no re-labelling, and only the new rubric costs anything:

$ langchef calibrate diff
  kappa  +0.57 -> +0.84   +0.27 [+0.11, +0.45]   IMPROVED
  moved  0 miss(es) fixed, 0 introduced; 8 false alarm(s) fixed, 0 introduced

That interval is a paired one, because both rubrics judged the same examples against the same labels. Agreement and kappa explains why that matters more than it sounds like it should.

Step 7 — Have the experiment designed · 1 minute

You do not have to work out how many examples you need, or what size of change counts as real. Describe what you are doing and let the tool cost it:

$ langchef experiment design     --intent "move to the replacement model, quality must hold within 3 points"     --variant-arm new-model --kind non-inferiority --margin 0.03

2 candidate design(s) for support
 -> as-it-stands   n=90     detects >=13.2%  53 judge call(s)
      note: These goldens cannot resolve 3.0%. The smallest effect this
            design could detect is 13.2%.
    powered        n=1745   detects >=3.0%   1655 judge call(s)
      note: Needs 1655 more golden(s) than the suite has (90).
  This is a proposal. Nothing runs until: langchef experiment approve ...

That note is the answer most tools will not give you. Ninety examples cannot resolve a three-point change — and you now know it before spending anything, rather than after a run comes back inconclusive. Collect more goldens, or accept the detection limit deliberately.

If you are trading quality for cost or latency, --margin is not optional: it is how much you are willing to lose, and it has to be set now. Deciding it after seeing the result is how a null result quietly becomes a green light.

langchef experiment approve support-new-model   # a person, on the record

The design lands in evals/experiments/ as reviewable TOML with a content hash. Edit any part of it afterwards and the approval lapses by itself, which is what stops an experiment being reshaped around the result it produced.

Step 8 — Run it and read out · 1 minute

langchef baseline set                       # pin the model you're on today

# re-answer the same questions on the replacement model, into
# evals/goldens/support.new-model.jsonl, then:

langchef judge run --arm new-model --experiment support-new-model
langchef experiment readout support-new-model

Running under the experiment holds it to the budget you approved: at the ceiling the run stops, exits 4, and writes exactly what it did not score — rather than quietly spending more than you agreed to. The readout refuses an unapproved design, and refuses a run that stopped short of the one you registered.

The questions must be identical across both arms — same example_id, different answers. That pairing is what lets a small number of examples say anything at all.

One of three things comes back.

  baseline 83.3%   variant 63.3%
  difference -20.0% [-27.8%, -12.2%]  p=0.0000
  REGRESSION

It broke something. The range says the true damage is somewhere between 12 and 28 points.

  difference +0.0% [+0.0%, +0.0%]  p=1.0000
  INCONCLUSIVE
  (smallest effect this run could have seen: 6.0%)

This is the one people misread. It does not mean the change was safe. It means ninety examples could never have detected anything smaller than a six-point swing, so if your change moved things by three points, this run was always going to shrug. Add examples, or accept that you are not going to resolve a change this small.

Step 9 — Write it up · 5 seconds

langchef memo render

A one-page Markdown memo in evals/memos/. It opens with whether the judge can be trusted, because a confident result from an unchecked judge is worse than no result. Then the finding, then what the run could not rule out. Commit it next to the change it justifies.

After the first time

Steps 1–6 are setup. From then on, each change costs you two commands and about a minute:

langchef judge run --arm my-change
langchef compare --variant support-my-change

Re-label every month or so, and whenever you touch the rubric or change the judge model — the judge's trustworthiness drifts as your traffic changes, and the tool will keep quoting the last calibration until you refresh it. If you are migrating the judge model itself, re-calibrate first: you are changing the instrument, not the thing being measured.

Handing it to an agent

If you use Claude Code, the adapter ships the whole playbook above as a skill, so the agent runs the loop and reads the results the way this page describes:

claude plugin install ./adapters/claude-code

The approval gate stays in the CLI rather than in the prompt, so it holds whether or not the agent cooperates.