Automated research systems are increasingly used for tasks such as literature synthesis, ideation, running experiments, writing, and peer review. But according to a new survey on arXiv, the way these systems are evaluated is highly fragmented. The authors note that evaluation is spread across many tasks, benchmarks, and individual studies, making direct comparison difficult.
The paper positions itself as a review of current benchmarks and evaluation practices. Its aim is to map the landscape rather than propose a single new metric. By cataloging how automated research is assessed, the survey provides a reference point for researchers who need to choose or design evaluations.
The abstract does not detail specific findings or recommendations, so the article is limited to the stated scope and motivation. The key contribution appears to be the synthesis of dispersed evaluation methods into a single overview, which could help future work align on more comparable standards.