Automatic prompt optimization (APO) is a technique that uses feedback from model responses to improve prompts without human intervention. Typically, its success is measured by overall scores, such as average accuracy across a benchmark. But a new paper on arXiv argues that this aggregate view is misleading: it does not show which individual questions are solved, lost, or unchanged after optimization.
The authors highlight that two prompts with the same overall score can behave very differently at the item level. One might fix many easy questions while breaking a few hard ones; another might make steady but unremarkable progress. Without looking at item-level transitions, researchers cannot tell whether optimization is genuinely improving a model's capabilities or merely shifting errors around.
The paper therefore calls for a shift in how APO is evaluated. Instead of relying solely on aggregate accuracy, it suggests examining the transitions of individual items across optimization steps. This finer-grained analysis could reveal patterns that overall scores obscure, such as which types of questions are consistently improved or degraded, and could lead to more targeted prompt optimization strategies.