A new preprint on arXiv takes aim at how model coherence is measured. The authors argue that much existing research treats output variance as evidence of inconsistency, without ruling out other explanations. They single out two: ambiguity, where a question admits multiple valid answers, and indifference, where the model has no strong reason to prefer one answer.
To address this, the paper introduces a set of 175 questions. The title points to a specific failure mode: narrow finetunes contradict themselves under resampling. That is, models tuned for narrow tasks can give different answers when the same question is resampled, but the paper cautions that such variance is not automatically a contradiction.
Because the abstract is truncated, details of the question set and experimental results are not fully available in the source. The paper's contribution, as described, is to distinguish genuine incoherence from benign variation.