AI labs have built their reputations on benchmarking, but Hugging Face's chief policy officer, Irene Solaiman, cautioned at TechCrunch Disrupt that the field is far from settled. She said reproducibility is a core problem: without access to the model itself, results are hard to verify, which is where open weights become important. She also noted that evaluations "start to get ineffective relatively quickly" and that there is no thriving ecosystem of standardized tests.
Solaiman pointed to the growing push for embedded evaluators—third-party observers placed inside AI organizations—as a response to safety incidents, but she questioned whether the qualifications for such evaluators are well defined. The skepticism is not hypothetical: Anthropic's decision to hire Accenture as its first evaluator drew public mockery over whether a consulting firm was suited for the task.
She also described a shift in focus from evaluating models alone to evaluating the entire system, including the harness and agent. A weaker model with a better harness can outperform a frontier model, she said, yet there are far fewer evaluations for harnesses. "Agentic evals are really hard and really expensive," Solaiman added, noting that the industry still lacks robust tools for holistic agent assessment.
The overall picture, she suggested, is a free-for-all. If benchmarking is to remain useful, the industry needs a more uniform approach where tests are comparable, reproducible, and tied to clear standards—something that does not yet exist.