What does FairStress reveal about how language models actually decide between people?

A benchmark built on pressure not just preference

The dataset contains 300 realistic two-candidate decision scenarios across 29 domains including hiring, healthcare, lending, education, housing, and criminal justice. Some scenarios have a clearly stronger candidate whose profile better satisfies the stated criteria. Others are deliberately ambiguous so there is no objectively better choice. The benchmark also varies how explicitly demographic identity appears — from no information to a coded name, an implied identity, or an explicit label. This design lets researchers test whether demographic differences emerge only when identity is stated outright or also when it is conveyed subtly.

Each decision is challenged with adversarial prefixes. These statements draw on authority, social consensus, purported evidence, legal concerns, historical disadvantage, and other forms of persuasion. One example: a senior leader has privately reviewed the files and recommended Candidate B for this role. Every prefix has a matched version that makes the same kind of case for the other candidate. Combining the variations in demographic identity and prefix expands the 300 scenarios into more than 13 million test cases. A 50,000-item subset called FairStress-Core uses item response theory to make evaluation faster while retaining much of the full benchmark's ability to reveal differences.

Models do not hold correct answers equally

When prompts contained no demographic information, models selected the clearly stronger candidate at nearly the same rate across otherwise equivalent versions. On ambiguous cases the choices were also close to evenly split.

As demographic identity became more explicit, behavior became less symmetric. Suppose Candidate A is clearly better qualified than Candidate B. When A was majority-coded — carrying the conventionally advantaged identity such as white, male, or heterosexual — and B was minority-coded, a prefix favoring B was more likely to make the model switch from the stronger candidate to the weaker one. When A was minority-coded and B was majority-coded with qualifications held fixed, the same kind of prefix favoring B was less likely to make the model abandon its correct choice of A.

The authors describe this pattern as anti-stereotypical over-correction. Across 15 models evaluated on the full benchmark, the average gap in abandoning correct answers was 6.9 percentage points with implied identity and 9.3 points with explicit labels. On FairStress-Core, frontier models including Claude Opus 5 and GPT-5.5 selected the minority-coded candidate about 60 to 65 percent of the time when neither candidate was objectively stronger. Under matched adversarial prefixes, switching toward minority-coded candidates was 8.8 to 42.4 percentage points more common than switching toward majority-coded candidates across 12 model-and-mode runs.

Explanations also differed. Across 11 open-weight models, explanations for minority-coded choices used somewhat more confident and positive language. When identity was explicitly labeled, these explanations contained words such as resilience, diversity, and barriers at higher frequency. Explanations for majority-coded choices included more procedural terms such as evidence, criteria, and detailed. A higher selection rate for minority candidates does not necessarily mean fairer treatment. When a model foregrounds identity or diversity in explaining a well-supported choice, it risks downplaying the achievements and qualifications that justify selecting that candidate.

Internal activations tell a different story

In a controlled analysis of Gemma-3-12B-it and Qwen2.5-7B-Instruct, internal activations predicted which demographic side the model selected with about 73 percent accuracy. Text descriptions of those same activations reached 57 to 59 percent against a 50 percent chance baseline. Removing candidate names lowered activation-based accuracy by about 18 percentage points. The activation analysis is limited to two models and only a handful of demographic contrasts. The authors treat these findings as narrower than the broader behavioral results. Still, they suggest that a model's reasoning trace may not reveal everything represented internally when it makes a decision.

Why existing benchmarks can produce a different result

The team compared FairStress results with BBQ, a widely used benchmark that tests whether models favor stereotype-consistent answers. In the BBQ evaluation, models generally leaned toward stereotype-consistent answers while FairStress revealed a tendency to favor minority-coded candidates. To understand the difference, they examined how the questions were asked and the answers scored. The standard BBQ setup scores probabilities of fixed answer options without having the model generate a response. FairStress asks the model to generate its own response and choose between the candidates. They took the same BBQ questions and model weights and tested five configurations that varied the chat template, question wording, and use of fixed-option probability scoring or answer generation. With a chat template, yes/no wording, and generated answers, Qwen2.5-7B-Instruct, Gemma-3-27B-it, and Llama-3.1-70B-Instruct shifted from stereotype-consistent to minority-favoring behavior. Llama-3.1-8B-Instruct moved to roughly neutral. The direction of measured bias depends partly on how a benchmark asks the question and scores the answer. Relying on fixed-option scores alone could miss over-correction in generated decisions.

Training data shows a gap between refusal and merit

To investigate possible sources of the asymmetry, the team audited data from Olmo 3's training pipeline. Its supervised fine-tuning corpus includes about 99,000 examples from two refusal-training datasets. Across 28,000 topic clusters in the fine-tuning and preference-training data, they found only nine clusters discussing decisions based on merit rather than demographics — and all nine consisted of refusals. The audit also found unequal portrayals of demographic groups. Minority-coded groups received more protective or positive framing. White and cisgender people received little protective framing and persistent negative framing. These findings suggest a gap between teaching models to refuse discriminatory requests and showing them how to make decisions on merit. Whether these patterns cause the behavioral gaps remains an open question.

Most fixes do not close the gap

Nine interventions were tested, including self-critique, activation steering, and learned edits. Most methods did not reliably reduce the asymmetry. Some produced more balanced choices at a substantial cost to accuracy. One activation-steering method reduced Qwen2.5-7B-Instruct's accuracy on clear-cut decisions from 98 percent to 47 percent. One approach came closer to symmetric robustness: counterfactual chain-of-thought, which asks the model to reconsider a case with the candidates' demographic identities swapped. Across the three model families tested, this approach reduced the gap in how readily models abandoned correct decisions while largely preserving accuracy. It did not remove every asymmetry. On ambiguous cases, models still favored minority-coded candidates.

What this changes for model evaluation

A model's demographic fairness cannot be judged solely by whether it reaches the same final choice across otherwise equivalent scenarios. A model can select the candidate best supported by the evidence in two versions of the same scenario that differ only in demographic identity, yet be easier to sway in one version, explain its choice differently, or show different internal activation patterns. Under FairStress's equality-centered definition, fair behavior means applying the same criteria and defending a well-supported decision consistently regardless of which demographic group benefits. The benchmark is available for download and is being built into Olmo-eval, the open workbench for running and comparing model evaluations. The dataset is published as allenai/fairstress.