Can a Small Local Model Decide When to Think Harder and When to Ask for Help?

Why Local Models Need a Traffic Controller

Running a language model on a laptop or a home server gives three clear wins: privacy, low latency, and no per‑token bill. The hardware that fits in a house, however, only holds small models. Those models make more mistakes than the frontier systems hosted in the cloud. The usual fix — sending the hard queries to a cloud model — throws away the privacy and cost advantages that made local deployment attractive in the first place.

A deployment that stays local faces two distinct decisions whenever a query looks difficult. First, the system can spend extra computation on that query, for example by reasoning step by step before answering. That raises accuracy but adds latency, so the system must decide which queries are worth the extra work. Second, some queries are simply beyond the local model's ability. Delivering a wrong answer is worse than handing the query to a human in the loop, so the system must decide which answers are safe to deliver.

Two Signals From Inside the Model

The paper shows that both decisions can be read from the model's own hidden states. The prefill state, computed before any token is generated, predicts whether the model will answer correctly. The answer state, captured at the end of the generated answer, predicts whether that answer is correct. HARISSA fine‑tunes the model so that both states become reliable predictors of correctness.

No external verifier is needed. No second model is loaded. The same weights that produce the answer also produce the confidence signals. This matters because every extra model in memory reduces the size of the primary model that can fit on the device.

How the Cascade Chooses the Cheapest Safe Answer

HARISSA builds a single policy that cascades through ways of answering from cheapest to most expensive. It starts with a plain forward pass. If the prefill state predicts that this pass will fail, the policy skips it and moves to the next method — for instance, a short reasoning trace. If that method's prefill state also predicts failure, the policy tries a longer reasoning trace. When the policy finally stops at an answer, the answer state checks it. If the answer state predicts the answer is wrong, the query is deferred to a human.

The cascade never runs a method that the prefill state has already flagged as doomed. It never delivers an answer that the answer state has flagged as unsafe. Both filters operate on the same model, at inference time, with no round‑trip to a server.

What the Benchmarks Show

On a device running a single model, HARISSA stays within one accuracy point of chain‑of‑thought at 2.7 times lower latency. On a server that holds four sizes of the same model family, HARISSA is more accurate than the FrugalGPT and Self‑REF cascades at the same latency. At the same deferral rate, the answer state leaves fewer wrong answers than standard confidence signals in five of six task and setting pairs.

Experiments do not report how many of the tested queries were actually deferred to humans. It also does not say how much fine‑tuning data or compute HARISSA requires compared with the base model.

Where the Paper Stops Short

The experiments use a single model family across four sizes. Whether the same prefill and answer states transfer to other architectures or to quantized versions is not tested. The latency numbers assume the cascade runs on one device; network overhead for the multi‑model server setup is not broken out. The human‑in‑the‑loop deferral path is simulated, not measured with real users.