A new arXiv preprint argues that conventional AI benchmarks are poorly suited for measuring future systems. Because these benchmarks are calibrated against human performance or today's AI models, they either become circular—testing what AI already does—or saturate once models exceed the reference point. The authors propose an alternative: an LLM-referenced C-test, a fill-in-the-blank style exam where the difficulty is anchored to the language model itself rather than to humans.
The paper, titled "Artificiality's First Exam," frames this as a way to evaluate machines that may become more capable than any current reference system. By using the model's own performance as a moving baseline, the test could remain meaningful even as AI advances beyond human-level ability. The authors call this target 'supraintelligence,' distinguishing it from benchmarks designed to compare against human baselines.
Since only the abstract is available, the full methodology is not yet detailed. The core claim, however, is clear: evaluation needs to escape the human-centric calibration that makes existing tests obsolete. Whether the C-test format can deliver on that promise remains to be seen, but the motivation addresses a real and growing problem in AI measurement.