harness-tuner
Measure your agent harness. Find where it wastes the model. Prove the fix worked.
Harness agnostic, agent agnostic, machine agnostic. Set it up by saying one sentence to your coding agent.
What it does
Your agent harness is the scaffolding around the model: the system prompt, the tool set and how the tools are described, the memory and context policy, the hooks, the retries, the compaction. Most of the difference between a harness that feels sharp and one that feels expensive lives there, and almost none of it is measured.
harness-tuner measures it. It reads traces from any harness through a small adapter, computes a fixed set of operational metrics, tells you where the harness is spending the model's capability on nothing, and writes proposals you can apply. Then it does the part nobody else does: it tells you, with a real statistical guarantee, whether the change you applied actually helped.
It never edits your harness. It proposes; you decide.
Install: pick a rung
Work down this list and stop at the first that fits your machine. Machine agnostic needs more than one answer, because a laptop, a locked-down build box and a CI runner are genuinely different problems.
Container
Needs no runtime at all. The conformance suite runs at image build time, so a broken image is never published.
Reference implementation
Python 3.11 or newer, standard library only. No install step, no dependencies, no build.
Reimplement, then prove it
Implement the protocol in whatever your machine has, then pass the shipped conformance fixtures. Tested, not trusted.
docker run --rm -v "$PWD:/work" -w /work ghcr.io/csnyder256/harness-tuner:latest doctorThen copy AGENT-GUIDE.md into your project and tell your coding agent:
Follow the agent protocol, and set this up for our harness.It reads your harness, works out what it can and cannot see, asks you the questions only you can answer, and writes the adapter. There is no SDK to learn and no integration to maintain.
The loop
Three claims, and what keeps each one honest
Every evaluation tool makes these claims. The interesting question is what stops each from quietly becoming false, because all three fail silently and none of them fails loudly.
1. It measures your harness
How that goes wrong: a metric the adapter could not observe gets reported as zero. A harness whose adapter cannot see caching then looks identical to one that genuinely caches nothing, and the tool has invented a finding out of a gap in its own instrumentation.
What prevents it: a value that could not be observed is null, and null becomes unavailable carrying a reason you can act on. Never zero, never an estimate. A metric that is undefined rather than merely unobserved also says which, so a recovery rate over zero errors reports undefined instead of a perfect score. The report leads with what could not be measured, and doctor is a whole command devoted to listing what your install cannot do.
2. It tells you what to change
How that goes wrong: a model is handed a trace and asked to find problems. It finds problems. It would find problems in any trace, and none of them come with a step number.
What prevents it: detection is arithmetic, not judgement. Every finding is produced by counting things in the trace and carries the exact steps it came from. Proposals carry a bound computed from what happened, not a forecast: at most nine of these reads were avoidable, because each had already been performed earlier in the same task.
3. It proves the change helped
How that goes wrong: you run the harness twice, the second number is better, and you believe it. Agent runs are noisy enough that this is close to a coin flip. The usual fix is a fixed sample size and a p-value computed once at the end, which fails here because trials cost money and you are going to look at the running result whether or not the method permits it.
What prevents it: an anytime-valid e-process. Each paired trial is a bet on the variant, and Ville's inequality bounds the probability that the accumulated wealth ever crosses the threshold, at any stopping time, checked as often as you like. verify reports an e-value, never a p-value, and returns confirmed, refuted, or unproven.
What it deliberately does not do
It publishes no numbers about anybody's harness
There is no leaderboard here and there never will be. Rankings between products go stale within weeks, invite methodology fights, and are almost always run by someone with a stake. Every number this tool produces is produced by you, on your machine, about your harness.
It does not apply its own proposals
A tool that can change the thing it measures is a tool that can make its own reports look good.
It does not call a model by default
The engine writes a request, your agent answers it with whatever provider and keys you already have, and the engine reads the answer back as a file. No credential store and no vendor SDK to go stale.
It does not skip tasks on a guess
A task your harness structurally cannot run is skipped and counted, never scored as a failure. With no capability scan present, nothing is skipped at all, because silently shrinking the task set would inflate every rate computed from it.
It has no opinion about your containment
Drive mode executes your harness with its real tool access, so it insists you record what that can reach and approve it. It does not insist on Docker, or a VM, or anything else, because it cannot know what your machine supports.
HTP-1
Everything else is negotiated with you at setup. Three things are fixed, and they are what make two runs comparable and a reimplementation checkable.
How it compares
| What it is | How harness-tuner relates | |
|---|---|---|
| Braintrust, Langfuse, LangSmith | Hosted LLM observability and output scoring | Excellent at tracing and grading what your agent produced. They are products with a control plane; this is a file you copy in that publishes nothing and phones nowhere. |
| DeepEval, Promptfoo, Ragas | Open-source scoring libraries for LLM output | They score answers. This scores the scaffolding: rework, loops, cache behaviour, context growth, recovery. Complementary rather than competing. |
| AgentBench, SWE-bench, tau-bench | Academic benchmarks with fixed tasks and leaderboards | They rank models on shared tasks. This ranks nothing, and generates a task set for what your harness is actually for. |
| OpenTelemetry, OpenInference | Tracing conventions for LLM applications | Genuinely adjacent and a fine source for an adapter to read. They standardize how a span is shaped; HTP-1 standardizes what a harness must expose to be evaluated. |
| Your own eval script | The thing most teams actually have | Usually the right starting point, and usually missing unavailable-versus-zero discipline, a test that survives being watched, and a way to prove a change helped. |
Questions
- Does this work with my harness?
- Yes, and no harness is named in the code. The adapter your agent writes is the only harness-specific part, and its whole job is to emit one JSON file per task. If a harness leaves any record of what it did, down to a plain text log, it can be measured, and the report says exactly what that costs you in metrics.
- What if my harness exposes almost nothing?
- Then roughly a third of the metrics report unavailable with a reason each, and
doctortells you before you run anything. Loop detection, step counts and tool-sequence analysis all still work, and those are frequently where the largest recoverable waste is. - Do I have to run real, expensive tasks every time?
- No. Record a driven run once and every later run replays from the cassette, free and byte-deterministically. That is what makes results reproducible by a colleague with no API key, and what makes this practical in CI.
- Why an e-value and not a p-value?
- Because you are going to look at the running comparison, and with a p-value looking and then continuing to gather data inflates the error rate silently. An e-process removes the problem instead of asking you not to peek, and it stops early when the answer is obvious.
- Is the fingerprint just vibes?
- It is a stated function of measured values, and the scales are published in the artifact next to every score they produced. A dimension whose inputs were all unavailable renders as insufficient evidence rather than falling back to a middle score, and a run with too few tasks renders no scores at all.
- Is there a leaderboard?
- No, and there will not be one.
Related
RAG-OS, a blueprint for a personal always-on AI agent. org-memory-os, one shared permission-aware AI memory for an organization.