Researchers who audit large language models for brand recommendations often send the same prompt multiple times to see how outputs vary. But there is no agreed-upon way to decide how many repetitions to run or how to choose other parameters, making results hard to compare across studies. A new arXiv paper, "The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations," highlights this gap.
The paper proposes a standardized protocol called the Dice Roll Method, which aims to give researchers a consistent framework for setting iteration counts and other aspects of repeated-query audits. The abstract notes that no such protocol currently exists, suggesting the method fills a methodological void.
By standardizing the process, the method could make audits of LLM brand recommendations more reproducible and comparable. The paper is available on arXiv under identifier 2609.04047v2.