Can a vision-language model run faster on a phone without retraining?

Why aggressive pruning breaks the model

Large vision-language models turn an image into a sequence of tokens — often hundreds of them — before the language backbone starts reasoning. Feeding all those tokens to the LLM is slow, especially on a phone or an embedded board. Researchers have tried dropping tokens early, but the model gets stupid fast. The paper identifies three distinct reasons.

First, most selectors look at the text prompt to decide which image tokens matter. That creates a bias: tokens that look irrelevant to the current question get discarded, even though they might matter for the next question or for a different reasoning path. Second, every dropped token carries visual information that is gone forever. Third, when you remove tokens from the middle of a sequence, the remaining ones shift positions. The LLM expects a fixed spatial layout; the shift scrambles the positional encoding it learned during pretraining.

These three effects compound. A method that solves only one still fails under heavy compression.

Three fixes that run before the LLM

VPRune adds a lightweight module between the vision encoder and the LLM. It has three stages, each targeting one of the failure modes.

Visual-only diversity selection picks a representative subset of tokens using only the image embeddings. No text prompt participates, so the selection cannot overfit to the current question. The algorithm measures pairwise distances in the embedding space and keeps tokens that cover the visual space broadly.

Similarity-guided token recycling rescues information from the discarded set. For every dropped token, the method finds its nearest neighbor among the kept tokens and adds the dropped token's residual — the difference vector — to that neighbor. The kept token count stays the same, but the aggregated vector carries more of the original image.

Position-preserving restoration writes the selected and recycled tokens back into the original sequence length. Empty slots are filled with a learned placeholder that carries no visual signal but preserves the positional indices the LLM expects. The LLM receives a full-length sequence with the same positional map it saw during pretraining.

All three stages are deterministic, require no gradient updates, and add negligible compute.

What the benchmarks show

The authors test on FastVLM-1.5B, a compact LVLM designed for edge deployment. They evaluate across multiple vision-language benchmarks — the paper does not list them by name — and report a favorable accuracy-compression trade-off. The advantage grows as the compression ratio increases. At aggressive token reduction, VPRune degrades less than baselines that use text-guided selection or simple top-k pooling.

Latency measurements run on an edge device — the exact hardware is not specified in the abstract. End-to-end inference time drops while task performance stays superior to the compared methods. The paper frames this as evidence of practicality for resource-constrained LVLM deployment.

No training runs, no fine-tuning schedules, and no hyperparameter sweeps are described in the abstract. The method is presented as plug-and-play for existing models.

Where the explanation stops

The abstract does not say how many tokens FastVLM-1.5B normally produces, what compression ratios were tested, or which specific benchmarks were used. It does not name the edge device, report absolute latency numbers, or compare against methods that do require training. Whether VPRune works on larger models like LLaVA-7B or on video inputs is not addressed. The code and model weights are not linked in the abstract.