Multilingual language models are often audited for bias by comparing average stereotype scores across languages. A new arXiv paper challenges this practice, arguing that such comparisons may be fundamentally misleading.
The authors note that typical audits compare a model's average stereotype score in English with its average in another language, then read the difference as a difference in fairness. The paper asks directly: how much of that difference is actually meaningful? The abstract is truncated mid-sentence, so the full findings are not yet visible, but the framing suggests that a lower bias score in translation does not automatically mean a fairer model.
This matters because real-world deployments increasingly rely on multilingual models, and fairness claims based on simple score averages could lead to incorrect conclusions about where bias actually persists.