How does STEPQuant compress recurrent states without losing accuracy?

Why recurrent states resist naive quantization

Linear attention avoids the ever-growing key-value cache that plagues standard transformers. Instead it keeps a fixed-size recurrent state that updates at every decoding step. Under concurrent serving those states occupy a substantial portion of GPU memory. Quantizing them to low precision seems like an obvious fix. Direct quantization however causes severe accuracy degradation. The reason is simple. Quantization errors do not stay isolated. They propagate through successive state updates and accumulate across the entire generation. A single corrupted value keeps influencing predictions long after it was written.

What makes errors stick or fade

The authors identify two complementary dimensions that determine error impact. Temporally, errors persist across many decoding steps. Spatially, different key rows affect outputs differently.

State magnitudes also vary substantially along both rows and columns. Uniform precision ignores both facts. It wastes bits on harmless entries and starves the sensitive ones. The paper shows that a single corrupted value keeps influencing predictions long after it was written. This temporal persistence combines with spatial sensitivity to make uniform quantization dangerous. STEPQuant is a spatial-temporal post-training quantization framework built for Delta-rule recurrent states. It allocates precision according to error magnitude and memory lifetime. The method jointly fits key-row and value-column scales based on state distributions and key-row impact on output error. No retraining is required. The framework operates entirely after the model is frozen. Precision follows the error profile. Bits go where they matter most.

Results on large models

Experiments run on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct.

Benchmarks cover both long- and short-generation tasks. Under a nominal 6-bit budget STEPQuant closely matches FP32-state accuracy. In a 4-bit configuration it outperforms uniform INT8. The paper reports over 5x recurrent-state compression with 6-bit STEPQuant. Total serving memory drops by up to 68.7 percent. These numbers come from the arXiv preprint dated 29 Sep 2026. The quantization integrates into SGLang with optimized GPU kernels. The kernels execute the non-uniform scaling without slowing down the attention loop. Code is available at the project repository. Nine authors contributed: Bingchen Yao, Haobo Xu, Haokun Lin, Yichen Wu, Ziyu Guo, Renrui Zhang, Zhichao Lu, Zhenan Sun, Ying Wei. The work appears as a technical report spanning computation and language, artificial intelligence, and machine learning.