Why do looped models still need separate memory for each pass?

A new method called BeNice! lets looped neural networks reuse the same memory buffer across repeated passes through their layers. The approach cuts memory growth that normally scales with each recurrence.

How looped models accumulate memory

Looped architectures reuse a single block of layers many times. Each pass through that block produces its own key-value cache during inference. Memory therefore grows in direct proportion to the number of recurrences. Earlier work noticed that the final recurrence's cache can sometimes stand in for the earlier ones without hurting quality. The authors call this property KV-shareability. Why it appears in some trained models and not others has stayed unclear. Even identically configured runs do not consistently show it.

The two-stage fix: stochastic depth then BeNice!

The paper proposes a deliberate path to shareability. First, stochastic depth during pretraining pushes the residual dynamics toward contraction. Representations across recurrences converge toward a common limit. Second, a post-training stage named BeNice! broadcasts the final recurrence's cache into the first recurrence. The first recurrence learns to emit residuals that remain compatible with that shared cache. Validation perplexity matches standard fine-tuning that keeps caches separate.

What the numbers show in practice

Simulated low-memory long-context runs reveal concrete gains. Cache hit rates climb by up to 31.4%. Throughput rises by up to 23.6%. The 90th-percentile time to first token drops by up to 83.6%. These figures come from the paper's own experiments.

Where the method fits with other compression tricks

KV-sharing composes with orthogonal techniques. The authors list multi-latent attention and Kimi delta attention as sequence mixers that benefit. They also list fp8, mxfp8, and nvfp4 as low-precision quantization formats that stack with the shared cache. The paper does not claim these combinations are exhaustive.

What remains unanswered

The work does not measure how many existing looped models already exhibit shareability without intervention. It also does not report wall-clock latency on specific hardware. The submission date is 22 Oct 2026.