Can Cheaper AI Benchmarks Still Tell You If a Model Is Biased?

The benchmark is the measurement instrument

A benchmark looks like a test. In practice it is a measurement instrument. When you swap the protocol — batch size, precision, number of questions — you are not just saving compute. You are changing the ruler. The paper submitted on 31 August 2026 treats that swap as an intervention and asks whether the readings stay the same. Three models were evaluated. Two are dense. One is a mixture-of-experts system. The benchmarks are BBQ and BBQ-V, both designed to surface social bias. The baseline is the full benchmark run in BF16. Seven conditions were compared against that baseline.

What the seven conditions actually are

Larger batch sizes. INT8 quantization. INT4 quantization. Reduced benchmark subsets. Combinations of the above. Each condition was measured for accuracy, bias severity, bias prevalence, reasoning quality, subgroup behavior, subset-membership stability, runtime, and measured GPU energy. The authors did not stop at aggregate accuracy. They looked at whether the same examples flip labels, whether the same subgroups look worse, whether the same reasoning patterns appear.

Batching looks like a free lunch — until it isn't

Batching keeps accuracy within 0.35 percentage points of baseline. Subgroup changes are comparatively small. Energy drops in five of six model–dataset settings. That sounds like a free lunch. It is not free for every model. The mixture-of-experts system behaves differently on BBQ-V. The paper does not say why. It reports the divergence and moves on.

Quantization does not preserve what you think it preserves

INT8 largely preserves quality. The phrase "largely preserves" hides a range. Measured GPU energy rises to 1.79–4.26× baseline. The hardware runs longer or hotter. The paper does not explain the mechanism. It reports the numbers. INT4 causes larger, model- and context-dependent changes. Bias severity shifts. Subgroup rankings shift. Subset-membership stability drops. The same example can move from biased to not biased depending on whether the model runs in BF16 or INT4. That is not a rounding error. It is a different measurement.

Smaller benchmarks save the most compute

Reduced benchmarks provide the most consistent savings. Very small subsets are substantially more sensitive to which items are retained. Drop a different 10 percent of questions and the bias prevalence number moves. The paper does not give a threshold where stability returns. It shows the curve and notes the sensitivity.

The conclusion you draw does not survive the reduction

Runtime drops. Energy drops. The conclusion you draw about a model's bias profile does not survive the reduction. The authors argue that efficient evaluation should be treated as a measurement intervention whose validity must be checked across the conclusions the benchmark is intended to support. They do not say who should check it. They do not say how often.

What this changes for anyone running evals

If you run BBQ or BBQ-V on a schedule — weekly, nightly, per release — you use batching or quantization or a subset to keep the bill down. The paper tells you that the numbers you log are not directly comparable to the full BF16 run. The difference is not noise. It is systematic. It varies by model architecture. It varies by benchmark version. The project website and code are linked in the paper. The authors do not provide a drop-in correction factor. They provide the raw comparisons. You decide whether your pipeline needs a re-baseline. The paper does not answer that for you.