Researchers have released Frame Evidence, a benchmark designed to separate two factors that can change a spatial answer: the reference frame implied by language and the visual evidence available in an image. The dataset includes 960 Chinese multiple-choice questions derived from 320 matched triplets, using both captured and rendered scenes. One set of questions compares three grounded reference frames over the same images; another pairs an image deletion (annotated as insufficient) with an addition that preserves the answer.
On 288 held-out questions across 20 configurations, GPT models reached 50% item accuracy in a three-shot setting, but the best three-frame group accuracy was just 5 out of 48, or 10.42%, achieved by GPT two-shot. In other words, models often answered individual questions correctly but failed to stay correct when the same question was posed under all three reference frames. The authors also found that few-shot prompting and task-specific adaptation raised item accuracy without reliably improving joint correctness, and repeated runs showed answer instability beneath similar average scores.
Frame Evidence is intended as a diagnostic rather than a leaderboard: its joint metrics require correct answers across all three frames and paired evidence variants, so a model cannot mask inconsistency by averaging. The authors note that input controls reveal where images support matched success, pointing to specific conditions under which current vision-language models still lack robust spatial reasoning.