How does ResidualQuant shrink KV cache for looped transformers to 2-bit residuals?

Why looped transformers still hit a memory wall

Looped transformers reuse the same block of parameters over several recurrent passes, adding computational depth without adding weights.

Every pass writes its own key and value tensors into the cache, so memory grows linearly with the number of loops. Large batch inference quickly exhausts GPU memory, capping throughput.

Standard KV quantization compresses those tensors to lower bit-widths, but at aggressive settings — INT4 or below — quality drops sharply. The paper notes that existing rotation-based methods still lose noticeable accuracy when pushed to INT2.

The residual idea

The authors observe that KV states from one loop to the next are highly similar. ResidualQuant picks the final-loop KV tensors as a reference and stores only the difference for earlier loops. Those residuals are small in magnitude, so they survive extreme quantization.

Measured gains on an RTX 5090

The pipeline adds three ingredients. Least-square scaling finds an optimal per-channel factor before rounding. Rotations spread quantization error across dimensions. Loop-wise mixed precision assigns higher bits to loops that matter more. Together they enable INT2 residuals with efficient reconstruction.

Theoretical KV storage falls by 80.7 percent. At the same memory budget the method beats the rotation baseline by up to 13.0 percent accuracy on math reasoning and code generation benchmarks. Fixed-batch decode throughput rises up to 2.73×. The smaller footprint allows batches up to 2× larger, pushing peak throughput up to 4.15×.

The paper does not report latency for single-request streaming. It also does not compare against non-looped models of equivalent depth. How the technique behaves on longer context windows or different loop counts is not specified.