Yesterday I wrote about GEPA which is a prompt optimizer that uses an LLM to read execution traces and evaluator feedback and rewrite one prompt at a time in a agentic system. By default, GEPA rotates through prompts , while choosing the candidate to mutate using an instance-level Pareto frontier over validation examples.
In comparison AgentGrad another prompt optimizer focuses on two parts of multi-agent prompt optimization: deciding which agent to change and combining feedback from multiple failures.
It works in four steps.
a) For each failed example, AgentGrad performs sequential changes in reverse execution order. It adds a hint derived from the ground-truth answer to one agent at a time and reruns the downstream agents. The first intervention that makes the final system output correct identifies the agent whose prompt should be optimized.
b) It compares that agent's original output with the output produced under the intervention. This difference is given to a LLM which converts the into a sample-level textual gradient describing how the prompt should change.
c) AgentGrad groups similar gradients into semantic minibatches. An aggregator LLM converts each cluster into one what they call as generalized textual gradient (prompt change) representing a common failure pattern.
d) A prompt optimizer generates a candidate prompt change edit from each generalized gradient. The edit is accepted only if it improves the corresponding semantic minibatch and also improves or maintains performance on the validation set.
Observations
a) GEPA chooses which prompts to optimize using round-robin selection. AgentGrad instead chooses individual agents and tests whether changing that agent's behavior improves the end-to-end failure.
b) AgentGrad in my opinion changes local behavior and then tries to generalize it by grouping local changes.
c) GEPA maintains multiple candidates rather than one evolving prompt set using Pareto frontier mechanism. It would be interesting to combine this with AgentGrad's approach.