How does SQuaT remove the accuracy floor in label-free quantized distillation?

Quantization-Aware Training lets developers shrink a neural network to low-bit precision while keeping most of its accuracy. The catch: the process assumes you still have the original labeled dataset.

That dataset is often gone.

Privacy rules, copyright claims, or simple cost make it unavailable. Knowledge Distillation has been the standard workaround — a large teacher model guides a small student using only unlabeled data. Prior work combining Quantization-Aware Training with Knowledge Distillation hits a hard limit.

During distillation, the teacher produces full-precision activations. The student operates with quantized weights and activations. Their numerical ranges do not align. This range mismatch induces an unattainable residual. The distillation loss cannot drop below an irreducible lower bound. No amount of tuning removes it. The paper states this as a fundamental limitation of earlier approaches.

SQuaT stands for Student-Aware Quantized Teacher Features.

The framework applies the student's own quantization parameters to the teacher's features during distillation. In effect, the teacher is forced to speak the student's numerical language. By quantizing the teacher's intermediate representations with the student's scale and zero-point, the range mismatch disappears. The distillation target becomes reachable. The theoretical lower bound on the loss is eliminated.

The approach is label-free. It requires no ground-truth annotations. It works as a drop-in modification to existing Quantization-Aware Training pipelines.

Experiments cover diverse settings. The paper reports consistent improvement over strong baselines. The gains are particularly pronounced in extreme low-bit regimes — specifically 1-bit and 2-bit configurations. At higher bit-widths the gap narrows but remains measurable.

The method does not rely on specific architectural assumptions. Evaluations span multiple model families and quantization recipes. Broad applicability is claimed and demonstrated. Source code is released alongside the paper. The work was accepted at AISTATS 2026.

The paper does not report wall-clock training time or memory overhead. It does not compare against full-precision distillation with labels available. The practical impact on inference latency for specific hardware targets is not measured. These questions remain for follow-up work.