How does a single pass fix quantized vision transformers when data shifts?

Why quantized ViTs break under shift

Post-training quantization shrinks a vision transformer so it fits on edge hardware. The process freezes the quantizers — the lookup tables that map floating-point activations to low-bit codes — using statistics collected on the original training set.

When the model later sees images from a different domain, the activations no longer fill those frozen ranges the same way. The code distribution drifts. Accuracy drops.

Existing test-time adaptation methods were not built for this constraint. Most recover accuracy by backpropagating through the network. That requires gradients, memory for the computation graph, and time. Backprop-free alternatives add extra forward passes or update running statistics inside the model. Lightweight feature- or logit-level tricks recover only part of the loss. None of them directly correct the mismatch at the quantizer input.

What QuAR does at the quantizer boundary

Quantizer-Aligned Recalibration (QuAR) operates on the activations right before they enter a frozen quantizer. It tracks the running per-channel mean and variance of the incoming test stream. It then shifts and scales those activations so their statistics match the calibration statistics that the quantizer expects.

No gradients flow backward. No model parameter changes. No second forward pass. The paper describes a single fixed configuration that stays constant across all test conditions.

Results on ImageNet-C with ViT-B

At 8-bit weight and activation precision QuAR reaches the highest mean accuracy among state-of-the-art backprop-free TTA methods. It beats the strongest baseline by 2.28 points. At 3-bit precision the gap widens to 4.00 points. The method also holds the lead at 4-bit and 6-bit settings.

Latency drops 46 percent compared with the strongest baseline. Memory overhead measures 0.17 MB — 0.01 percent of peak inference memory.

Why the code distribution matters

The authors trace the gain to a reduced per-channel mismatch at the quantizers. When the input statistics are realigned, the quantizer emits codes that match the distribution seen during calibration. Baselines either leave that distribution unchanged or distort it further.

A single fixed configuration stays ahead across continual streams, non-i.i.d. label shift, seven out-of-distribution suites, and three other backbones. The paper spans 44 pages with six figures. Code is available at the repository linked from the arXiv entry.