Language models are increasingly built around skills—reusable instructions and procedures that guide task execution. This approach lets models apply structured knowledge rather than reasoning from scratch each time. But a new paper from arXiv highlights a failure mode that arises when such skills meet external information.
The authors describe a scenario where a model first selects a relevant skill, then must combine that skill's governing procedure with retrieved content. When the retrieved information only partially conflicts with the skill's expectations, the model may produce a procedural hallucination—an output that follows the skill's structure but diverges from the retrieved facts in problematic ways.
The study focuses specifically on partial conflicts, where the mismatch is not total but subtle enough to trip up the model. This suggests that simply improving retrieval quality may not be enough; models also need to handle cases where retrieved evidence and learned procedures are not perfectly aligned. The paper is available on arXiv under the title "When Retrieval Partially Conflicts with Skills: Understanding Procedural Hallucination in Language Models."