Can a 4-bit model beat its own 16-bit version?

A compressed language model runs on fewer bits. Usually that means worse answers. Multiverse Computing says their new healing method flips that relationship.

The standard pipeline for shrinking a large model has three stages. First, cut the architecture — remove layers, heads, or neurons. Second, quantize the remaining weights to 4 bits. Third, heal the damage. That last step is where most of the argument lives.

Why the usual healing hits a ceiling

Quantization-aware training (QAT) inserts fake-quantization operators into the forward pass and keeps fine-tuning on a task loss. It works, but it re-runs the entire post-training history — supervised fine-tuning, RLHF, agentic tuning — through a noisier forward pass. The process is expensive and, as the authors show, unstable if training runs past its sweet spot.

A different approach, quantization-aware distillation (QAD), avoids that history. It freezes a full-precision teacher and distills into the quantized student via KL divergence on logits. This works when the only change is precision. Once the model has been structurally compressed, however, there is no independently trained full-precision version of the smaller architecture. The only candidate teacher is the recovered bfloat16 checkpoint — itself a distilled approximation. Distilling from it anchors the student to a degraded target.

How Quantization-Aware Healing changes the teacher

QAH removes that ceiling with one change: it distills directly from the original, pre-compression model. The teacher is full-size and full-precision. The student is half the size and runs in MXFP4. They do not share an architecture. Because a teacher's output distribution is architecture-agnostic, the size mismatch does not block the transfer. The student never sees hard labels, only the teacher's logits matched through KL divergence.

This reframes what quantization does. Under QAH it is no longer a lossy post-processing step applied after healing finishes. It becomes a second, full pass of distillation against the original teacher — supervision the bfloat16 checkpoint never received. The 4-bit student is not compensating for information lost to quantization. It is picking up information the earlier recovery stage did not have the time or data to transfer.

Stability falls out of the loss itself. KL distillation ties the student to a fixed teacher distribution. Once the student catches up, there is no further pressure to drift. A cross-entropy task loss keeps pushing toward hard labels indefinitely. That difference shows up in both accuracy and training stability.

The benchmark table

The authors applied QAH to a GPT-OSS 120B model compressed to 60B parameters, recovered in bfloat16, then re-quantized to MXFP4. The natural comparison is against that same 60B model's bfloat16 checkpoint.

The QAH model wins on 7 of 9 benchmarks. The two losses are under 1.5 points. The largest gains land on the capabilities compression usually damages most: long-context reasoning and math.

Against the original 120B teacher, the QAH model surpasses it on LiveCodeBench (66.5 vs 66.0) and comes within 1.6 points on GPQA Diamond. The largest remaining gap is on AA-LCR, where the capacity lost to compression is intrinsically hardest to recover.

Speed and stability head to head

To isolate the loss function, the team compared QAH against QAT on a GPT-OSS 9B model quantized to MXFP4, tracking average performance across MMLU-Pro, LiveCodeBench, and GPQA Diamond.

Both methods reach a similar peak — 54.9 for QAH, 54.6 for QAT. The difference is in the path. QAH hits its peak in roughly 100 steps. QAT needs about 700. After peaking, QAH stays within two points of its best. QAT collapses, shedding nearly 19 points by step 1,200.

A QAT checkpoint needs careful early stopping against a held-out signal to avoid shipping a model that has already started to degrade. A sufficiently trained QAH checkpoint can be served safely because it simply does not drift.

What this changes for deployment

At 4-bit precision the QAH model uses roughly four times less weight memory than the bfloat16 student. At half the parameter count of the 120B teacher it roughly halves compute per token, which lets it run on substantially smaller hardware. For model families that ship in bfloat16 rather than 4-bit, the combined parameter and precision reduction would be closer to eight times less compute per token.

Quantization stops being a tax you pay for efficiency and becomes an extra opportunity to teach the model. The healing pipeline reaches that point in a fraction of the training a QAT recipe would need.

The work sits alongside a companion paper on efficient distillation that supplies the long-context training machinery QAH depends on — specifically a chunked KL-divergence loss that computes KL one sequence slice at a time and never materializes the full vocabulary-by-sequence grid, making 32k-token healing fit inside a fixed GPU memory budget. How far the recipe generalizes across other architectures and compression ratios is not addressed in the announcement.