Standard hallucination stress tests can give the same average score for different language models, but that similarity can be misleading. According to a new arXiv preprint, the same test may prompt hallucinations in one model while another model fails on entirely different inputs, even though both end up with comparable overall numbers.
The authors propose a procedure that iteratively edits source-grounded seed prompts to build stress tests that are target-aware. The idea is to move beyond one-size-fits-all benchmarks and toward evaluations that expose which specific prompts cause a given model to fabricate information.
The abstract itself cuts off mid-sentence, so the full method and results are not available from the source. Still, the motivation is clear: average scores can mask model-specific weaknesses, and better tests may need to be tailored to each model rather than shared blindly.