Guide

Bring your own data

You already have a labelled test set in a spreadsheet. Point at it, name the columns, and start. No trace collection, no rubric writing, no labelling session.

The short version

Start here is the path for a team whose data is still inside a running app: collect traces, assemble them, write a rubric, label forty examples. That is an afternoon of work before the tool has told you anything. If your data already exists as a file, this page is a much shorter road, and for three of the four task classes it takes minutes.

Declare it once, in the workspace

The column mapping lives in evals/config.toml, not on the command line:

[dataset]
path  = "data/support-tickets.parquet"
class = "classification"
input = "ticket_body"
label = "resolved_category"

That is deliberate. Which column is the input and which is the target is a claim about what your data means, and it belongs somewhere a colleague can disagree with it in a pull request. As a flag it would be invisible and unversioned, and the next person would have no way to tell what the last run actually measured.

CSV and Parquet both work. The file stays where it is; nothing is copied into the workspace.

The class decides almost everything

Class	A row is	Needs a judge?	Calibration applies?	Setup
`qna` / `generation`	question → golden answer	yes	yes	an afternoon
`classification`	text → ideal label	no	no	minutes
`retrieval`	query → relevant doc ids	no	no	minutes
`reranking`	query + candidates → ordering	no	no	minutes

An unknown class is refused, and the message lists what is available. It does not fall back to qna, because that would quietly attach a judge and a calibration to data that has a hard answer, and produce numbers that mean nothing.

The honest part: calibration does not apply to three of these

LangChef's usual pitch is the judge you can trust. That only makes sense when the target is free text and something has to decide whether an answer is good. A classification label, a relevance judgement and an ideal ordering are hard targets. There is nothing to calibrate.

What you get instead is the other half, and it is a real offer: the paired comparison, the detection limit, the refusal to call an underpowered result a pass, and the experiment discipline. Those are worth having. They are not the same pitch, and saying they were would be dishonest.

So on those three classes, calibrate, taxonomy and label plan refuse rather than returning a number nobody can interpret.

What gets compared

Each class has a per-example outcome, and the comparison follows its shape:

Class	Per-example outcome	Comparison
`qna` / `generation`	the judge says pass or fail	exact McNemar, paired
`classification`	`predicted == ideal`	exact McNemar, paired
`retrieval`	recall@k, MRR, nDCG	Wilcoxon signed-rank, paired
`reranking`	nDCG, MAP	Wilcoxon signed-rank, paired

Classification is not reduced to pass or fail. "Correct or not" is the comparison you want, and McNemar on it is exact, so nothing is thrown away.

Retrieval is not thresholded. This was the decision worth arguing about. Recall 0.42 and recall 0.71 are both "fail" against a threshold of 0.8, and the difference between them is the entire finding. So retrieval scores stay continuous and get a comparison of their own. How that works.

What is not padded, guessed, or quietly dropped

A row that cannot be read is reported, never dropped. Silent row loss changes the denominator of every statistic downstream. A run over 900 of a thousand rows that reports 900 is lying by omission, and nothing later can detect that it happened.
A misnamed column names itself, and lists what was actually found. Not a KeyError. You should not have to guess which of your columns was wrong.
recall@10 from six documents scores the six. It is not padded to ten, because that would penalise your retriever for a truncation you chose.
A query with no relevant documents scores nan, not zero. That query cannot be measured, and a zero would drag your average down with a number that means "we could not ask".

Adding a task class

Task classes live in a pack manifest, not in the statistics core. Adding one is a directory: declare its schema, its metric set and its outcome shape in pack.toml, and the loader resolves it. Nothing in src/langchef/core/ knows any class by name, and a test fails the build if that ever stops being true.