Semantic query engines rely on large language models to judge whether individual rows match a query. These engines are typically benchmarked by running them a handful of times at temperature 0, on the assumption that the results are stable enough to reflect true quality.
A new paper on arXiv directly tests that assumption by measuring how much another execution changes the outcome. The authors focus on run-to-run variance in semantic query processing, asking a simple but important question: if you run the same engine again, do you get the same answer?
The study highlights a potential blind spot in current evaluation practice. If outputs vary across executions even at temperature 0, then benchmark scores based on a few runs may not be representative, and comparisons between engines could be misleading. The paper does not propose a fix, but it makes a case for reporting variance alongside quality metrics.