Bug bounty platforms receive vulnerability reports faster than vendors can triage them, and large language models are increasingly being considered for assigning CVSS severity scores. A new arXiv preprint, 'Narratives Mislead: An Empirical Study of How LLMs Misjudge CVSS Severity in Bug Bounty Reports,' investigates a potential weakness in this approach: the narrative text that accompanies each report.
The study empirically examines how LLMs judge severity when reports include narrative descriptions. According to the abstract, these narratives can mislead the models, causing them to misjudge CVSS severity. The exact experimental setup and quantitative results are not detailed in the abstract, but the title and framing make the central claim clear: the way a report is written matters.
The practical implication is that automated triage systems relying on LLMs may need to be designed with narrative bias in mind. If models are swayed by framing or storytelling rather than purely technical details, their severity ratings could be unreliable. The paper thus highlights a caution for using LLMs in bug bounty workflows.