Most agent benchmarks give AI systems a well-defined problem, such as a bug report or a specification. The new FDE-Gym benchmark argues that this misses a central part of real-world deployment: the initial request is often vague and requires back-and-forth with the customer.
The benchmark is named after forward-deployed engineers, who work directly with customers to turn fuzzy needs into actionable tasks. FDE-Gym appears to simulate that clarification process, testing whether an agent can ask the right questions and converge on a solution rather than simply executing a given instruction.
The abstract is brief, so details on the benchmark's tasks and evaluation are not yet available. But the motivation is clear: if AI agents are to handle the "last mile" of delivery, they need to be measured on ambiguity, not just on cleanly specified tickets.