How does GradCodes fine-tune quantized models without leaving low-bit space?

Why continuous proxies drift from the deployed model

Fine-tuning a quantized model usually means optimizing a full-precision copy and then rounding the result back to low-bit codes. The paper identifies two failure modes. Straight-through estimation error accumulates because the gradient flows through a continuous relaxation that does not match the discrete quantization function. Post-quantize gap appears when the rounded weights no longer sit at the optimum found in continuous space. Both effects grow larger as the bit-width drops.

Discrete search avoids these gaps by operating on the codes themselves. The cost is combinatorial explosion. Under a finite training budget the search cannot explore enough candidates to converge. The authors frame the problem as optimization over quantization codes and scales rather than over floating-point weights.

Code surrogate gradient as a first-order signal

GradCodes introduces a surrogate gradient defined directly in the quantized code space. The gradient points toward better codes without ever leaving the discrete domain. It is computed by differentiating a smooth surrogate loss that upper-bounds the true discrete objective. This gives a first-order signal that respects the deployment constraint from the first step.

The surrogate is constructed so that its minimizer coincides with the optimal code assignment. No continuous proxy weights are maintained. Memory stays at the low-bit level throughout training. The paper reports 408 KB for the full manuscript including figures and supplementary material.

Guided search keeps the checkpoint faithful

A pure gradient step in code space can still propose invalid code transitions. GradCodes adds a guided search layer that accepts or rejects each proposed code change based on the true discrete loss. The search uses the surrogate gradient as a proposal distribution. This preserves deployment faithfulness while accelerating convergence compared to unguided discrete optimization.

The combination means every intermediate checkpoint is a valid low-bit model. There is no final rounding step that could degrade accuracy. The method works across different quantization datatypes without modification.

Results

Experiments cover arithmetic reasoning, instruction following, and structured language understanding. GradCodes consistently improves fine-tuning low-bit models across different quantization datatypes. The paper does not report how many models were tested or which specific bit-widths were used. It also does not compare training time against full-precision fine-tuning baselines. Code is provided at a URL listed in the paper. The submission date is 31 Aug 2026. The authors are Shiguang Wu, Zhouchen Lin, and Quanming Yao. The DOI is 10.48550/arXiv.2608.30908.