How does Dual-QK compress KV caches to 2-bit while keeping accuracy at 128K context?

Why the cache becomes a bottleneck

Long inputs and extended generation increase the storage and access costs of the key-value cache. Every token adds new keys and values that must be kept for later attention steps. The cache grows linearly with sequence length. Low-bit quantization reduces storage and memory traffic. Query-channel pruning can further reduce key-cache reads. Rotation-based quantization redistributes the energy of key outliers across channels. To maintain computational invariance the same orthogonal transform must be applied to queries. This preserves query-key dot products. But the rotation disperses query energy. The separation between a few large components to retain and many small ones to prune weakens. Pruning becomes less effective. The cache stays large.

The conflict is fundamental.

Paired transforms instead of shared rotation

Dual-QK uses paired non-orthogonal query and key transforms. The method combines partial key whitening with a query-aligned basis. Calibrated query and key statistics guide the design. This balances key scales for INT2 quantization. It also concentrates query energy for dynamic channel pruning. Channel-0 protection and bucket-relative RoPE support low-bit accuracy over long contexts. The transforms are not orthogonal. They do not need to be inverses. Each serves a different purpose. The key transform flattens the distribution for quantization. The query transform sharpens the distribution for pruning. The conflict that rotation creates disappears.

Results at 128K context

Experiments ran on four models. They covered five generative benchmarks. Long-context retrieval tasks were also tested. Accuracy improved over OSCAR on most tasks at 40% query-channel sparsity. At a 128K context Dual-QK provides 6.8× KV-cache compression. An estimated 8.3× reduction in KV read volume appears relative to unpruned BF16. The paper does not say how many tokens were generated per request. It does not report memory usage in gigabytes. It does not compare against KV cache eviction methods.

Throughput on SGLang

The SGLang implementation achieves up to 3.75× the decoding throughput of unpruned BF16. The paper does not specify the hardware used for throughput measurement. It does not give latency percentiles. It does not show throughput at different batch sizes. The speedup comes from reduced memory traffic and compute. Pruning removes 40% of key channels from reads. Quantization cuts storage to 2 bits per value. Both effects compound.