Meta-Harness
This is a continuation of my thread on agentic system optimization.
In this one I want to show how Meta-Harness works in practice.
Quick recap of the setup.
A harness, as we know, is the code and tools around a fixed model. It decides what to store, what to retrieve and what to show the model at each step. The optimizer here is called the proposer.
It is a coding agent (Claude Code with Opus 4.6) that reads past attempts and writes new versions of the harness. Every past attempt is saved as a folder with its code, its score and its execution traces.
Now to the part of the paper I found most interesting.
This is a log from a 10-iteration run on TerminalBench-2, which is a benchmark of hard command-line tasks. The starting harness scored 64.4%.
a) The proposer read a lot before it changed anything. In each iteration it opened between 69 and 99 files, 82 on average. About 41% of those were the code of earlier attempts and 40% were execution traces. Only 6% were score files.
b) In iterations 1 and 2, it made two bug fixes. The first one removed leftover terminal markers that were confusing the agent and sending it into loops. The second one fixed a completion check that made the agent re-verify its work for 15 to 40 extra steps after it was already done. Both attempts also came with a new prompt. Both got worse, scoring 58.9% and 57.8%.
c) In iteration 3, it compared the two failed attempts and looked at what they had in common. The bug fixes were different, but the new prompt was the same in both. It figured out that the new prompt was making the agent delete files it still needed, and that the bug fixes had been tested together with this bad change. So it went back to the original prompt and tested only the bug fixes. That attempt scored 63.3%, just 1.1 points below where it started, which showed its reasoning was right.
d) After that, it kept trying smaller fixes. It finally landed on a safer change that added new behavior without removing anything, and that became the best harness in the run.
This is how I would want an engineer on my team to debug. Two changes fail, you look at what they share, you pull that part out and you test again.
This example shows the key principles of Meta-Harness.
a) Keep the full history. Every past attempt, score and trace is saved. Without the two failed attempts, the proposer could not have spotted what they had in common.
b) Give the proposer raw traces, not summaries.
c) Let the proposer decide what to read. The folder is too big to fit in one prompt, so the proposer searches it and opens only what it thinks matters.
d) Keep the search loop simple. There is no fixed rule for which earlier attempt to build on.