A new preprint on arXiv examines how large language models represent vulnerable code internally. The authors analyzed eight code models ranging from 6.7B to 15B parameters, using three datasets of C programs. Their focus was not just whether models can spot vulnerabilities, but what form that knowledge takes in the models' internal representations.
The study's title points to the central finding: models may "know" about vulnerabilities in a way that linear classifiers cannot easily access. The authors suggest that vulnerability information is encoded through relational structure rather than as a simple linear feature. This would explain why linear probing—a common method for inspecting model representations—can fail even when the model itself performs well on vulnerability detection.
The work highlights a gap between model capability and interpretability. If vulnerability knowledge is distributed across relations among code elements, then techniques that assume linear separability may underestimate what code LLMs understand. The abstract does not specify the exact datasets or evaluation details, but the implication is that more expressive probing methods may be needed to surface this knowledge.