Reference

Label planning

Which forty examples a person should spend their ten minutes on, why it is not a random forty, and the open question about what those labels can then be used to claim.

The worked example

Ninety scored examples. The judge failed fifteen of them. You have time for forty labels.

  langchef label plan --budget 40

  selected 40 of 90
    fail stratum   15   (every one the judge flagged)
    pass stratum   25
  19 of the 40 chosen because the judge was unsure

Why not just take forty at random

Because of where the information is. Pick forty of these ninety at random and you would expect about 6.7 of them to be examples the judge flagged. Your entire catch rate would rest on six or seven labels, and its interval would be so wide the number could not support a decision.

Stratifying by the judge's own verdict takes all fifteen instead. Same ten minutes, same forty labels, and the number you most need is measured on more than twice the evidence.

That gap widens sharply as suites get more realistic. On a production suite where the judge flags 2%, a random forty contains fewer than one flagged example on average. The catch rate is not merely imprecise, it does not exist.

Why the judge's uncertain cases come first

Within each stratum the plan sorts by the judge's own confidence and takes the least confident first. Nineteen of the forty above were chosen that way.

The reasoning is that a label is worth what it changes. A case the judge scored 0.98 will almost certainly agree with you, and confirms what you already believed. A case it scored 0.51 is where two candidate rubrics give different answers, so labelling it settles something.

This is uncertainty sampling, the oldest idea in active learning, and it is very good at the job it is built for: improving the instrument. Keep that phrase, because the rest of this page turns on it.

The same plan every time

Ties break on sha256(seed + example_id) rather than on list order or a random draw. So the same run and the same budget produce the same forty on any machine, in any order the scores arrived, and a plan can be regenerated without re-labelling anything. A labelling plan that shuffles when you re-run it is a plan nobody can pick up halfway.

The open question, and it is a real one

Each selected row carries a weight, the stratum size over the number taken. It looks like an inclusion weight, the kind that lets you scale a sample back up to the population. It is not one, and the rates on the calibration report are not population estimates.

The reason is on this page. Selection inside a stratum is not random: it takes the lowest confidence rows deterministically. So a row's chance of being picked is 0 or 1 given its confidence rank, not n/N. Post-stratification cannot repair that, because the selection tracks the very thing being estimated, and confidence tracks disagreement almost by definition.

A seeded coverage simulation contributed on #30 put a nominal 95% interval at 43% actual coverage for the catch rate, and at 100% for the true negative rate, which is the opposite failure and just as useless.

Read the calibration numbers as describing the forty examples you labelled, not as an estimate of your whole suite. The underlying question is whether one label budget can buy estimation and diagnosis at once, and it is open at #60. It probably cannot, in which case the plan splits into a random part for measuring and an uncertainty-selected part for diagnosing.

What the reasons mean

Every selected row says why it was picked, because a person labelling forty things deserves to know which ones are load-bearing.

Reason	Means
`judge was unsure`	Low confidence, and in the first half of its stratum. These are the ones most likely to change a rubric
`stratum coverage (fail)`	Here to keep the flagged stratum full, which is what holds the catch rate up
`stratum coverage (pass)`	Here to keep the false alarm rate measurable

Computed in src/langchef/core/sampling.py

plan() the split, the ordering, and the weight on each row

_tiebreak() hashed ordering, so the plan is stable across machines

summarise() what the plan did, for the person about to label

What we deliberately do not do

No adaptive re-planning mid-budget. The plan is computed once from the scores. Choosing the next example based on labels already given would be closer to real active learning and would make the sample even harder to reason about statistically than it already is.
No stratification on anything but the judge's verdict. Not topic, not length, not customer. Those are the slice dimensions the taxonomy reports along, and stratifying on the same axis you later test for concentration would bias the test.
No more than two strata. Verdicts are pass or fail here. Task classes with ordered or multi-class labels need a different design, which is part of the open work.
No estimate of how many labels you need. The plan spends the budget you give it. Whether forty is enough for the claim you want to make is exactly what #60 is about, and the honest answer today is that we do not know.