Reference

Designing a run

The experiment lifecycle in full: design, approve, check, read out. Every output on this page is real, captured from a workspace built from scratch, including the refusals.

Start here walks the whole product in nine steps and spends about a minute on this part. This page is the other half: what each of these four commands actually writes, what it refuses and why, and what the file on disk looks like at every stage.

Where this sits in a whole run

Designing is step nine of thirteen, and the eight before it are what make it mean anything.

	Command	What it is for
1	`langchef init`	Scaffold the workspace
2	collect examples	50–100 from real traffic
3	write the rubric	Judging and pins
4	`langchef approve rubric`	Gate one. Judging refuses at exit 2 without it
5	`langchef judge run`	Judging and pins
6	`langchef label plan`	Label planning
7	label them yourself	The only part nobody can automate
8	`langchef calibrate report`	Agreement, taxonomy
9	`experiment design`	From here down is this page
10	`experiment approve`	Gate two
11	`judge run --arm variant`	The other arm, same pin
12	`experiment readout`	Comparing two arms
13	`langchef memo write`	One page a person can disagree with

If your task has a hard target, steps 3 to 8 do not apply and the run is 1, 2, then 9 to 13. Which one am I?

Step 9 — design

You have 90 goldens, a scored baseline, and you want to move to the replacement model. You care about a five point change.

langchef experiment design --suite support --variant-arm variant   --intent "move to the replacement model" --target-effect 0.05 --id model-swap

Prose to stderr, for you:

2 candidate design(s) for support: move to the replacement model
 -> as-it-stands   n=90     detects >=13.2%    0 judge call(s)
      note: These goldens cannot resolve 5.0%. The smallest effect this
            design could detect is 13.2%; a real change below that will
            come back inconclusive.
    powered        n=628    detects >=5.0%   538 judge call(s)
      note: Needs 538 more golden(s) than the suite has (90). Collect them
            before running, or accept the detection limit of the design above.

The arrow marks the one recorded. --accept powered records the other. Neither is a refusal: choosing the 13.2% design knowingly is a fine decision, and choosing it without knowing is the thing this prevents.

What it writes

A TOML file in evals/experiments/, reviewed like code:

# LangChef pre-registration.
#
# Written before the run and reviewed like code. Editing anything below
# changes the digest and revokes the approval, which is the point: an
# experiment whose design moved after the traffic is not an experiment.

[experiment]
baseline_arm = "baseline"
variant_arm = "variant"
hypothesis = "move to the replacement model"
kind = "superiority"
n = 90
mde = 0.13206799371997818
target_effect = 0.05
power = 0.8
level = 0.95
discordance_assumed = 0.2
discordance_source = "assumed default (20%) — no prior comparison here"
rubric = "answer-quality@290335165c70"
stopping_rule = "Score all 90 goldens in both arms, then read out once. No
  interim looks: stopping early when a result looks good is the most common
  way an experiment reports an effect that is not there."
guardrails = [
  "The rubric must still be answer-quality@290335165c70 at run time; a changed rubric revokes approval.",
  "Both arms must be scored under the same pin, or compare exits 5.",
  "Calibration must exist for this judge; a memo without it says so in full.",
]

[budget]
judge_calls = 0
source = "the design's own estimate"

There is no [approval] block yet. That is the next step and a separate one.

It will not overwrite one

Run the same command with the same --id again and it refuses:

model-swap.toml already exists — pass a different --id, or delete it
deliberately. Overwriting a pre-registration silently is how one stops
meaning anything.

Step 10 — approve, which is gate two

Try to read out first and you get refused, at exit 2:

langchef experiment readout model-swap        # exit 2

langchef: refused — an approval gate is unmet — experiment model-swap has not
been approved — review the design, then run:
  langchef experiment approve model-swap

Approving appends a block and nothing else:

[approval]
digest = "3f1053d32846"
at = "2026-08-28T17:16:21+00:00"
by = "human"

That digest is a hash of the body excluding the [approval] block, which is what lets approval be recorded in the same file without changing what was approved.

Editing it afterwards revokes it, with nobody watching

Change one number in the design and leave the approval block alone:

-target_effect = 0.05
+target_effect = 0.15

langchef experiment readout model-swap        # exit 2

langchef: refused — an approval gate is unmet — the design for model-swap
changed since it was approved (3f1053d32846 -> 510086cece38). Re-read it,
then run: langchef experiment approve model-swap

It names both digests. You cannot widen a margin, drop a guardrail or move a target after seeing the numbers and still hold an approval. The whole mechanism is a hash over the body and about eight lines of code.

Step 11 — score the other arm

langchef judge run --suite support --arm variant --run-id var-1   --experiment model-swap

judge: n=90  provider_calls=24  cache_hits=66  cache_misses=24
       fail=42  pass=48  fail_rate=0.467  budget_exhausted=False  unscored=0

24 calls, 66 cache hits. Only the answers that actually changed needed judging, because verdicts are content-addressed on the example and the rubric. How the key works.

--experiment links the run to the pre-registration and applies its budget. Hit the ceiling and the run stops, writes what is left to runs/<id>/undone.json, and exits 4 rather than returning a partial result that looks whole.

Step 12 — read out

langchef experiment readout model-swap        # exit 0

readout for model-swap@3f1053d32846 on 90 shared golden(s)
  difference -26.7% [-35.6%, -17.8%]
  REGRESSION
  -> evals/runs/var-1/readout.json

The JSON an agent reads carries the working, not only the verdict:

{
  "experiment_id": "model-swap",
  "design_digest": "3f1053d32846",
  "baseline_run": "base-1",
  "baseline_rate": 0.8,
  "difference": -0.2666666666666667,
  "interval": {"lo": -0.3555, "hi": -0.1777, "level": 0.95},
  "discordance": {"broke": 24, "fixed": 0, "both_pass": 48, "both_fail": 18},
  "discordant": 24,
  "exploratory": false,
  "improvement": false,
  "inconclusive": false
}

design_digest ties the result to the exact design that was approved. The discordance block is the evidence: 66 goldens agreed under both arms and carry no information, 24 broke and none were fixed, and that is what the verdict rests on. Why only those 24 matter.

exploratory: true appears when a run departed from its design. The result is still printed, and it recommends no decision.

`experiment check` and `list`

check answers "does what I have match what was registered", and reports rather than decides:

langchef experiment check model-swap          # exit 0

model-swap@3f1053d32846: approved
  the run matches the pre-registration

langchef experiment list

1 pre-registration(s)
  model-swap                   approved   n=90

A known gap in check

Run check against an experiment whose variant arm was never scored at all and it still reports the run matches the pre-registration at exit 0, with variant_run: null in the payload. It is verifying the approval, not the existence of a run. Read variant_run rather than the sentence until this is fixed.

Why two candidates and not one answer

The first uses the goldens you have and says what they can resolve. The second appears only when you named a target the first cannot reach, and it is usually the more useful, because "you need 628 examples and you have 90" is an answer, where a shrug is not.

The failure without it is running the underpowered version, getting "no significant difference", and reading that as "no difference". Those are not the same sentence.

The number that governs everything

  detect 20.0%   ->  n =    40
  detect 10.0%   ->  n =   157
  detect  5.0%   ->  n =   628
  detect  2.5%   ->  n =  2512

Halving the effect you want to detect costs roughly four times the examples. That is the shape of the arithmetic rather than a property of this tool, and it is why "collect more goldens" stops being advice quite quickly.

It is computed from the discordant rate, not the pass rate, because the comparison is paired. With no prior comparison, 20% is assumed and discordance_source says so. Run the arms once and the next design uses your real rate: the example above came back with 24 discordant of 90, so the next design for this suite starts from 26.7% rather than the guess.

What a design fixes, and why each one

Field	Why it is decided in advance
`n`, `mde`	So "inconclusive" reads as "this could never have seen it" rather than "no effect"
`margin`	A tolerance chosen after seeing the interval is not a tolerance, it is a rationalisation
`stopping_rule`	Score everything, read out once. Stopping early when it looks good is the most common way an experiment reports an effect that is not there
`guardrails`	What must not get worse while you chase the thing you are chasing
`rubric`	Pinned by hash, so a changed definition of good revokes the approval
`cost`	Calls always, money only if a price was configured. Inventing a price is worse than admitting the gap

Non-inferiority refuses without a margin

a non-inferiority design needs --margin: how much quality you are
willing to lose. Deciding it after the run is not a design.

With a three point margin on these 90 goldens you get the same two-candidate shape and a second caveat worth reading twice:

[as-it-stands]  n=90     detects 13.2%
  ! Quality holds only if the whole interval clears −3.0%.
    The point estimate is not the test.

[powered]       n=1745   detects  3.0%

The gates, as exit codes

A rule in a prompt is a suggestion, and an agent can talk itself past one. These are exit codes.

Code	When	Meaning
2	no approved design, the design moved since approval, or more than one run matches an arm	Refused. Reading out whichever run ran last is how an experiment gets repeated until it says something better
5	the two runs were produced under different pins	Refused. You would be comparing two rulers, not two systems
4	the agreed call budget ran out	Stopped. What is unscored goes to `undone.json`

src/langchef/core/design.py — propose(), required_n(), estimate_cost()

src/langchef/workspace/experiments.py — the TOML, and digest() over the body without the approval

src/langchef/cli/design_cmd.py — design, approve, check, readout, list

src/langchef/core/gates.py — gate one on the rubric, gate two on the experiment

What we deliberately do not do

No sequential or always-valid designs. Score everything, read out once. Peeking and stopping when it looks good inflates the false positive rate badly. Always-valid confidence sequences are the real answer and are not built.
No multi-arm designs. Two arms at a time. Sweeping five variants and reporting the winner without correction finds effects that are not there.
No automatic approval. An agent can design and check. It cannot approve. That step is where a person accepts a cost and a claim, and automating it would remove the only human in the loop.
No cost model beyond calls times a price you supply. No token estimation, no provider price list to go stale. usd stays null until you configure a price.
No boolean for "is this design sensible". runnable_now says whether it can be executed with the goldens that exist and shortfall says how many more are needed. Whether 538 more goldens are worth collecting is a judgement, and the tool declines to encode one.