When people delegate multi-step work to an LLM agent, they typically read only the final reply the agent hands back. According to a new arXiv paper, existing evaluation metrics for such agents track whether the task was completed and whether the report is faithful to what the agent did, but they leave open a more basic question: can the requester actually understand and use the reply?
The paper, titled "Measuring and Reducing Unusable Replies of LLM Agents," proposes to treat "unusable replies" as a distinct failure mode and introduces a way to measure it. The authors note that a reply can be technically accurate yet still fail if the requester cannot make sense of it or act on it. They also present methods for reducing the frequency of such replies, though the abstract does not detail the specific techniques.
The work highlights a gap in how agent quality is assessed. If evaluation scores ignore whether the final output is comprehensible to the human in the loop, then improvements in task completion and faithfulness may not translate into better real-world outcomes. The paper's framing suggests that usability should be a first-class evaluation criterion for agent systems.