The abstract for AgentEval, posted on arXiv, challenges common assumptions in evaluating stateful agents. It states that task completion alone does not determine whether an agent satisfied its obligations, implying that success on a task can mask failures in meeting the underlying requirements.
The abstract also raises a second concern: a recorded judgment—such as a prior evaluation result—does not guarantee that the supporting evidence remains valid. This suggests that assessments can become outdated or unreliable if the evidence they rely on changes over time.
To address these issues, the authors present AgentEval, a system that carries a… (the abstract cuts off at this point). The available text indicates the system is designed for evidence-bounded assessment, but the full mechanism is not described in the source.