How Can Mobile NPUs Reuse KV Cache When Their Graphs Are Static?
On-device large language model serving gives users data sovereignty and freedom from cloud latency. A new paper shows how to make KV caching work on mobile NPUs despite their static computation graphs and tight memory bandwidth.
Why Cloud KV Caching Fails on Mobile NPUs
KV caching stores key and value tensors from earlier tokens so the model does not recompute them at every decoding step. On cloud GPUs the execution graph is dynamic: operators launch on demand, memory bandwidth is abundant, and the cache can grow or shrink freely. Mobile NPUs break every assumption. Their computation graphs must be fully compiled ahead of time. Memory capacity is measured in megabytes, not gigabytes. I/O bandwidth between DRAM and the accelerator is a hard bottleneck. The paper states these architectural assumptions do not hold on mobile NPUs. Local-first personal intelligence depends on solving this mismatch.
Intra-Graph Mechanism
The authors propose an intra-graph mechanism that maps selective KV recomputation onto the static NPU graph. Instead of treating the cache as a monolithic block, they identify which KV tensors can be reused across requests and which must be recomputed. The static graph then embeds both paths: a fast reuse path for prefix matches and a recomputation path for divergent suffixes. This reconciles algorithmic dynamicity with NPU staticity. The mechanism operates entirely within the precompiled graph — no runtime graph rewriting occurs.
Inter-Graph Scheduler
A second component schedules work across multiple static graphs. The inter-graph scheduler merges request chunks and minimizes padding via dynamic programming.
Hierarchical KV Manager
To address bandwidth limits the paper introduces a hierarchical KV manager. Its index combines a tree structure and hash lookups
The sources this article was written from
- "arXiv:2609.34727v1
https://arxiv.org/abs/2609.34727v1"