Reference

Commands

Generated from the contract inside the binary, so this table cannot claim a command that does not exist. 19 of 22 are live today. If you are here to learn the tool, start with your first evaluation instead.

The ones you will actually use

Command	When
langchef init	Once, in your project.
langchef approve rubric	After you write or edit the rubric.
langchef judge run --arm X	Every time you want a version scored.
langchef label plan --budget 40	Once at the start, then monthly.
langchef label import FILE	After you have filled the plan in.
langchef calibrate report	After importing labels. Tells you if the judge is trustworthy.
langchef calibrate diff	After changing the rubric. Tells you whether the change helped.
langchef baseline set	Once you have a version worth comparing against.
langchef compare --variant X	The question you came here to answer.
langchef memo render	To write the decision down.

Everything, generated from the contract

Determinism

deterministic — same inputs, same output, always. seeded — random but reproducible from a recorded seed. cached — results are keyed on content, rubric hash and model, so re-running an unchanged suite is free.

Command	Summary	Determinism	Writes	Status
langchef contract	Emit this contract as JSON	deterministic	—	live
langchef doctor	Verify environment, credentials, pins, budget	deterministic	—	live
langchef packs list	List resolvable expertise packs	deterministic	—	live
langchef init	Scaffold the workspace	deterministic	`workspace`	live
langchef approve rubric	Record a human approval of the rubric as it stands. Gate one	deterministic	`evals/config.toml`	live
langchef sample	Pull and stratify production traces	seeded	`runs/<id>/sample.parquet`	M5
langchef label plan	Choose the labelling subset that maximises information per label	deterministic	`labels/<judge>.todo.jsonl`	live
langchef label import	Ingest returned human labels	deterministic	`labels/<judge>.jsonl`	live
langchef judge run	Score examples against a pinned rubric	cached	`runs/<id>/scores.parquet`	live
langchef calibrate report	Agreement: TPR, TNR, confusion matrix, Cohen's kappa, disagreement taxonomy	deterministic	`runs/<id>/calibration.json`	live
langchef calibrate diff	Re-score a revised rubric against the same labels, report the delta	deterministic	`runs/<id>/delta.json`	live
langchef eval run	Run a suite over goldens	cached	`runs/<id>/`	M6
langchef baseline set \| show	Pin a run as the reference	deterministic	`baselines/`	live
langchef compare	Paired deltas, confidence intervals, regression flags, minimum detectable effect	deterministic	`runs/<id>/compare.json`	live
langchef triage	Slice drill-down, deploy correlation, reproduction set	deterministic	`findings/`	M6
langchef power	Is this many examples enough? Detection limit, required n, horizon	deterministic	—	live
langchef experiment design	Propose one or two designs with detection limit, sample size and cost	deterministic	`experiments/<id>.toml`	live
langchef experiment approve	Pre-register a design. Editing it afterwards revokes the approval	deterministic	`experiments/<id>.toml`	live
langchef experiment check \| list	Does the run match what was registered?	deterministic	—	live
langchef experiment readout	Gated readout. Refuses an unapproved design or an unfinished run	deterministic	`runs/<id>/readout.json`	live
langchef ledger append \| query	The persistent record	deterministic	`ledger/`	live
langchef memo render	Decision memo from run artifacts	deterministic	`memos/`	live

Exit codes

Code	Meaning	Status
0	ok	in use
1	unexpected error	in use
2	refused — an approval gate is unmet	in use
3	abstained — confidence below threshold	reserved — not yet emitted
4	budget exhausted; report of what was left undone written	reserved — not yet emitted
5	pin mismatch — judge model, version or rubric hash moved	in use

Two streams

Every command writes JSON to stdout for a program, and plain text to stderr for you. There is no --format flag; --help is the one exception, because it is written for people.

langchef calibrate report              # you read this
langchef calibrate report 2>/dev/null  # a script parses this

Other behaviour worth knowing

Workspace commands search upward for evals/config.toml, the way git finds a repository, so they work from anywhere in your tree.
--suite can be omitted when there is only one.
Each version's answers live in its own file: goldens/<suite>.<arm>.jsonl.
The rules an agent reads at runtime are in docs/AGENT-CONTRACT.md, or langchef contract.