Concepts

Every idea LangChef uses, in a few lines each, with a link to the page that explains it properly. Start here if a word in the output is unfamiliar, or if you are about to change the code and want to know what you are changing.

The things you supply

Example

One question your app was asked, the answer it gave, and the context it retrieved. Also called a golden. Fifty to a hundred is a normal starting set, taken from real traffic rather than invented, because invented questions are always easier than the ones users actually ask.

Rubric

What "a good answer" means, written down as Markdown. Each ### heading is one criterion and the judge must name the one it failed on. Roughly what you would tell a new teammate on their first day. It is hashed, so editing it revokes its approval. Why the headings matter.

Label

Your own verdict on an example, pass or fail, recorded by hand. This is the ground truth everything else is measured against. It is the only part nobody can automate for you, and roughly forty of them is the usual ask.

Judging

Judge

The thing that reads an answer and says pass or fail. Usually a model with your rubric in the prompt; sometimes, as in the default here, plain token matching. A judge is a measuring instrument, not a metric, which is the single idea the rest of the product is built on.

Pin

The rubric hash, the provider and the models that produced a set of verdicts, recorded on every run. Two runs are only comparable if their pins match. When they do not, compare exits 5 rather than drawing a chart of two different measurements.

Two-tier judging

A cheap model scores everything; a strong model re-scores only the cases the cheap one was unsure about. Built in from the start rather than added later, because the model is part of the cache key and retrofitting it would invalidate every cached verdict in every workspace. More.

Calibration: is the judge any good

Calibration

Comparing the judge's verdicts against your labels on the same examples, to find out how far it can be trusted. It comes before everything else because an eval suite built on an unchecked judge produces confident nonsense, and no downstream statistic repairs that. Full page.

Agreement, or Cohen's kappa

How much you and the judge agree, after subtracting the agreement two coins would produce by luck. Runs 0 to 1. It exists because raw accuracy flatters any judge on a lopsided suite: where 95% of answers are fine, a judge that says "fine" to everything is 95% accurate and worthless. How it is computed.

Catch rate and false alarm rate

Of the problems you found, how many the judge also flagged (catch rate, or TPR). Of the answers that were fine, how many it flagged anyway (false alarm rate, or FPR). These two say how a judge fails, which is what actually changes a rubric. More.

Confidence interval

The range the true value probably sits in. Printed beside every rate, because "80%" measured on fifteen examples and "80%" measured on fifteen hundred are different claims. A wide interval means label more examples, not that the judge is erratic. Why Wilson and not the textbook one.

Disagreement taxonomy

Six disagreements is a count, not a finding. The taxonomy groups them by which rubric criterion the judge cited and which slice of traffic they fell in, and refuses to report a slice whose interval does not clear the base rate. Full page.

Label planning

Choosing which examples are worth a person's ten minutes. Sampling at random on a suite where the judge flags 15% wastes most of the budget confirming passes, so the plan takes every flagged example plus a sample of the rest, and prefers cases the judge was unsure about. Full page.

Experiments: did the change help

Arm, run, baseline

An arm is one version of your app. A run is one scoring pass over one arm. The baseline is the run you pinned as the reference. Both arms answer the same questions with the same example ids; that pairing is what lets a small set say anything.

Paired comparison

Because both arms answer identical questions, only the examples that changed verdict carry information. If 200 pass under both and 3 flip, the evidence is in the 3. Treating the arms as independent samples throws that away and real regressions come back as noise. Full page.

Minimum detectable effect

The smallest change your run could have caught. Quoted on every inconclusive result, because "no significant difference" on ninety examples usually means "this set could never have seen it". It turns a shrug into a plan: collect more, or accept the limit knowingly. More, and how a run is designed around it.

Non-inferiority and the margin

Swapping in a cheaper model is not a hunt for an improvement, it is a check that the drop stays inside a tolerance you set before the run. You read the bottom of the interval against that margin, not the middle. Deciding the margin afterwards is how a null result becomes a green light. More.

Discipline

Pre-registration

The experiment design, written to evals/experiments/ and approved by a person before any traffic. It carries a content hash, so editing it afterwards revokes the approval on its own. Reading out without one exits 2; reading out a run that departed from one marks the result exploratory and recommends no decision. Full page.

Gates as exit codes

A rule in a prompt is a suggestion. These are exit codes: 2 refused because an approval is missing, 5 refused because the two runs were measured differently, 4 stopped because the agreed budget ran out. An agent cannot argue with a non-zero exit. The full table.

The quality ledger

An append-only record of every run, calibration and decision. Entries are never edited; a correction is a new entry, so what was believed at the time survives. It answers the question a leaderboard cannot: has quality moved this quarter, and what did we do about it.