What LangChef talks to
LangChef decides what to measure and whether the answer means anything. It is not trying to replace the tools you already run — if your runs live in MLflow, they stay in MLflow.
Shipped means it works today on main. Everything else is a
statement of intent and nothing more. Track the order of work in the
tracker; open an issue if you need one moved up.
| Integration | Kind | Status | What it does |
|---|---|---|---|
| MLflow | Experiment tracking | next up | Read runs and params from an existing MLflow server; write calibration and comparison results back as metrics and artifacts, so LangChef's numbers land where your team already looks. MLflow 3's align() is one of only two things in the ecosystem that automates judge-human agreement — interoperating with it beats competing with it. |
| litellm | Model providers | shipped | One shim over every provider litellm speaks — Anthropic, OpenAI, Google, Bedrock, Vertex, local. Install with uv sync --extra providers. Nothing else in the codebase imports a provider SDK, so this is the only file to rewrite if it goes bad. |
| Claude Code | Harness | shipped | The calibration playbook as a skill, plus commands that drive the CLI. The approval gates live in the CLI rather than the prompt, so they hold whether or not the agent cooperates. |
| Parquet + DuckDB | Storage & query | partial | Per-example scores are written as Parquet today. DuckDB as the read-side query engine over the workspace — never as the store — arrives with the connectors. |
| Langfuse | Tracing | planned | Pull production traces as evaluation examples instead of hand-assembling goldens. The obvious second integration: it is open source, self-hostable, and the topology matches — your traces stay where they are. |
| OpenTelemetry (GenAI semconv) | Tracing | planned | Reading traces through the OTel GenAI semantic conventions rather than per-vendor SDKs would cover several tracing tools at once. Worth doing before Langfuse-specific work if the conventions have settled. |
| Arize Phoenix | Tracing | considering | Open source, widely deployed, and its hosted sibling ships the closest competing agent. Reading from it is straightforward; the question is whether its users want this. |
| LangSmith | Tracing & eval | considering | Large install base among teams already on LangChain. Pulling datasets and traces out is feasible; writing results back into a vendor data plane is against the grain here. |
| Braintrust | Eval platform | considering | Overlaps rather than complements — its Loop assistant occupies the rung below this one. An import path for teams migrating is more plausible than a live integration. |
| GitHub Actions | CI | planned | Run the loop on a schedule in CI for teams without an agent harness, opening a pull request with the memo. The scheduled loop is the product; the harness is one way to run it. |
Why MLflow is first
Two reasons, and neither is popularity.
The first is where teams already are. A team running retrieval, ranking or a classifier alongside their GenAI feature almost certainly has an MLflow server, because the classical-ML tooling around it largely died off in 2025 and MLflow is what survived. Asking them to keep results somewhere else is asking them to maintain two records of the same thing.
The second is that MLflow 3 ships align() — one of only two things in the ecosystem
that automates judge-versus-human agreement at all. That makes it the nearest thing to a competitor
on the one problem LangChef leads with, which is exactly why interoperating beats competing:
a team that has already aligned a judge in MLflow should be able to bring it, not redo it.
What an integration is allowed to do
- Read-only by default. Connectors sample and read; they do not write to your production systems.
- No data plane. Traces are read where they live. Nothing transits infrastructure belonging to this project, because there isn't any.
- The workspace stays the record. An integration may mirror results outward — metrics into MLflow, a memo into a pull request — but the source of truth is the text in your repository. If you drop an integration, you keep everything.
Asking for one
The list is short on purpose and the order is not fixed. If a tool your team depends on is missing or sitting in "considering", say so on the issue tracker — what teams actually run beats what looks strategic from here.