Continuing my thread on agentic systems optimization. Harness optimization is all the rage so lets look at one technique. This one was specifically focussed on prompt and middleware optimization.
BTW, when I think of harness I include prompt, tool interfaces, state handling and recovery logic.
The optimizer, PRISM from a recent Airbnb paper , uses four techniques that I found useful.
It is a evolutionary technique on the lines of GEPA but adapted to harnesses. It keeps a Pareto frontier of harnesses during the search. A harness stays on the frontier if no other harness is at least as good on both measures (task and reliability e.g. not getting stuck in loop) and better on one.
a) It uses a three-way data split. One split is used to propose changes, a second is used to select the final harness, and a third is held out and used only to measure the final result. This separates the search process from the reported improvement.
Say there are 300 past customer support conversations where the agent failed. The first 100 go into the first split. The optimizer reads these failures and proposes changes to the harness, such as a new rule in the prompt. The next 100 go into the second split. Each proposed harness runs on these conversations, and the one with the highest pass rate is kept. The last 100 go into the third split and are not used during the search. When the search is done, the chosen harness runs on these 100 conversations once, and that is the reported result.
b) It measures worst-case lift in addition to average improvement by running the optimizer multiple times, selecting the harness that performs best on the validation split, and then measuring the 5th percentile improvement on held-out data
c)It limits middleware changes to a small set of patterns. Middleware sits between the model and its tools. It can fix malformed arguments before the tool runs. It can reject an invalid call and ask the model to try again. It can also block a call until a required step has happened.
For example, if the model sends expression but the tool expects math_expression, the middleware can rename the field before executing the call.
d) It routes failures to the layer that should fix them. PRISM clusters failed trajectories by root cause and decides whether the change belongs in the prompt, middleware, or both.
Where this could be improved so this can be more widely used
1. Only few benchmarks were covered.
2. The scope was limited to few middleware changes and feels like a proof-of-concept