According to a Hugging Face blog post from Ai2, the new FairStress benchmark is designed to measure demographic fairness in LLM decision-making beyond the final answer a model gives. It is available for download and is being integrated into Olmo-eval, Ai2's open evaluation workbench. The motivating problem is that standard fairness tests often swap demographic identities in otherwise identical scenarios and treat identical outputs as fair, potentially hiding deeper asymmetries in how models behave when pressed.
FairStress uses 300 realistic two-candidate decision scenarios across 29 domains, including hiring, healthcare, lending, education, housing, and criminal justice. Some scenarios have a clearly stronger candidate, while others are deliberately ambiguous. The benchmark varies how demographic identity is presented—from absent to demographically coded names, indirect implication, or explicit labels—and pairs each persuasive "adversarial prefix" with a matched version favoring the other candidate. That design expands the 300 scenarios into more than 13 million test cases, with a smaller FairStress-Core subset for faster evaluation.
The authors define their target as symmetric fairness: when evidence clearly supports one decision, models should be equally resistant to changing that decision regardless of which demographic group benefits. In experiments, they found little difference in behavior when prompts contained no demographic information. But across open-weight and frontier models and multiple training stages, the same kind of persuasive argument—say, a senior leader's recommendation or a claim about historical disadvantage—could make models more or less likely to switch decisions depending on which candidate it supported. They also observed differences in how models explained decisions and in internal activation patterns across domains. Since this is a single source, there are no conflicting reports to compare; the post itself notes that FairStress does not attempt to capture every conception of fairness, such as equity-based approaches that may intentionally favor historically underrepresented groups.