Did that change make it better or worse?

The situation

You maintain a retrieval app, a classifier, or an agent that calls a few tools. It works. Then:

• your provider retires the model you shipped on — you did not choose this, and you have weeks;
• you swap the embedding model, add a reranker, or change chunking, and retrieval shifts underneath the generator;
• you distil to a fine-tuned small model to cut the bill, where the question is not "is it better" but "did quality hold";
• you give an agent another tool, or someone edits a prompt.

Same question every time, and it usually gets answered by eyeballing twenty outputs, or by a pass rate in a spreadsheet that moved three points, or not at all — which is the most common answer, and why only about a third of teams running AI in production evaluate it on live traffic.

This is not a tooling problem. Half a dozen capable eval platforms exist and several are free. What is missing is the person who knows whether your grader can be trusted and how many examples a number needs before it means anything. LangChef stands in for them.

Which of these are you

The answer changes what this tool does for you, so it is worth settling before anything else. The difference is not what your app is built from. It is whether you already know the right answer.

Neither is the lesser path. If you have hard targets you have solved the hardest problem in evaluation already, and you get an answer sooner. What you still need is the part that has nothing to do with judging:

• Paired statistics. Both arms answer the same questions, so only the examples that changed carry information. Treating them as two independent samples is how a real regression comes back as noise.
• The detection limit. "recall@5 went from 0.71 to 0.69" means nothing without knowing the smallest move your set could have seen.
• The discipline. The margin decided before the run, the design approved before traffic, the refusal to read out whichever run happens to look best.

Four ways this goes wrong

The grader nobody graded

Your judge marks 95% of answers good, which sounds healthy. If only 5% of your answers are genuinely bad, a judge that marks everything good also scores 95% — and from that number alone you cannot tell the two apart.

LangChef: you label forty examples once. It then reports whether the judge agrees with you more than luck explains, how often it misses real problems, how often it cries wolf, and which kinds of answer it gets wrong.

The difference that isn't there

83% to 80% on ninety examples. That swing is comfortably inside what randomness produces when nothing changed — but the number moved, and somebody has to make a call.

LangChef: returns regression, improvement, or can't tell, with the range the true difference sits in. When it can't tell, it says how small a change this test set could ever have caught.

The ruler that moved

You tightened the grading prompt on Tuesday. Monday's 83% and Wednesday's 88% were never measuring the same thing, but they still line up on a chart.

LangChef: records which rubric and model produced every number, and refuses to compare two runs measured differently. It stops rather than drawing the chart.

Not knowing which half broke

Quality drops four points after an embedding swap. Is the generator worse, or is it being handed worse context? Those have nothing in common as fixes, and one pass rate over both cannot separate them.

LangChef: one rubric criterion per failure mode — grounding for retrieval, correctness for generation — and the judge must name the one it failed on, so failures stay attributable instead of pooled.

You are the diner, not the chef

The name is about who does the cooking. You say what you want in plain language. The harness — Claude Code, or whatever agent your team already runs — takes the order and comes back with one or two ways to test it. You pick one and agree what it may spend. The kitchen runs the sampling, the judging and the statistics out of sight. What arrives is a dish and one question: is this good enough to ship?

That question is the only judgement the design asks of you, and it is the only one that does not need an evaluation background. Everything before it — which metric, how many examples, whether the judge can be trusted, whether the difference is real — is kitchen work.

What you get back

What it is not, and who it is not for

It ships no test cases — your examples come from your traffic, because a benchmark of someone else's questions tells you nothing about yours. There is no UI. And somebody does have to say what "good" means and mark forty examples: anyone promising otherwise is selling you a judge nobody checked.

Skip it if you already have an eval team and a calibrated judge — you have solved this — or if your feature is still changing shape weekly. Come back when it settles.

It does not replace the tools you already run. If your runs live in MLflow, they stay in MLflow; LangChef is the thing that decides what to measure and whether the answer means anything. See integrations.