Researchers behind the new arXiv paper "Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?" ask a pointed question: when a self-driving system is given a natural-language instruction, does it actually follow it? The paper notes that recent vision-language-action (VLA) models increasingly use such instructions to condition their driving behavior.
The authors argue that current closed-loop benchmarks are not designed to answer that question. Instead, these benchmarks primarily measure route completion, safety, and rule compliance. That means a model could score well without ever truly adhering to a specific instruction, because the benchmarks do not directly evaluate whether the instruction was followed.
The implication is that the field may need new evaluation methods that directly test instruction following, rather than treating it as a side effect of good driving metrics. The paper is available on arXiv under identifier 2610.16634.