Command-line interface (CLI) agents are increasingly expected to handle long, multi-step tasks, but the benchmarks used to measure them do not reflect how people actually work. According to the abstract for the arXiv paper "RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents," existing benchmarks either assume user instructions stay fixed for the whole run or judge only the final outcome, ignoring what happens in between.
The paper argues that this leaves a gap: real CLI use involves ongoing human-agent collaboration, and evaluation should capture those runtime interactions. RealCLI is presented as a step toward a unified evaluation that does so, rather than treating the task as a one-shot instruction-following exercise.
Because the abstract is brief, the full methodology and results are not yet visible from the announcement. But the motivation is clear: as agents take on longer-horizon work, evaluation needs to move beyond static instructions and final outputs to include the collaborative process itself.