A recent arXiv paper raises a pointed question about Theory of Mind (ToM) benchmarks: do they actually measure social reasoning, or are they just testing parsing and algorithmic state tracking? The authors note that while existing benchmarks vary in modality and dialogue length, they may all re-encode a common set of simpler computational patterns.
If that is the case, then high scores on these benchmarks might not indicate that a model understands beliefs, intentions, or perspectives. Instead, the tasks could be solved by tracking states and following surface-level linguistic cues—skills that are impressive but not equivalent to theory of mind.
The paper's abstract is truncated in the source, so the full argument and evidence are not available here. But the central concern is clear: the field may need to reexamine what its benchmarks actually prove before claiming progress on social intelligence.