How does WUSH-KV shrink the KV cache without breaking attention?

Why the cache became the bottleneck

Every transformer layer stores a key vector and a value vector for each processed token. Contexts of tens or hundreds of thousands of tokens make those vectors dominate memory and bandwidth. Quantizing the cache seems obvious. Naive quantization breaks attention scores.

How WUSH-KV builds its transforms

The authors adapt WUSH, a technique originally designed for weight quantization. WUSH constructs a linear transform from the second-order statistics of both matrices in a matrix product. For keys this is the query-key product. For values it is the attention-weight-value product. Calibration data passes through the model once. The covariance of keys and the covariance of values are estimated separately. The resulting transforms rotate each space so a simple clipped affine quantizer introduces minimal error.

One transform offline, one at runtime

The value transform is absorbed into the output projection weights of the attention block. The model is modified once offline. Quantized values are written directly without extra multiplication at inference. The key transform cannot be folded because of rotary position embeddings. It is applied to keys after RoPE just before the quantizer.

What the numbers show

Layerwise reconstruction error drops compared with the OSCAR baseline. End-to-end perplexity reaches the lowest value among tested transforms. At two bits per component WUSH-KV matches or beats OSCAR across every model and task evaluated. The integration uses SGLang with an OSCAR-style percentile-clipped affine quantizer.

Gaps

The preprint does not report wall-clock latency or throughput measurements. It does not specify how many calibration sequences were needed. Stability under distribution shift is not addressed.