How does QATFactory train FP4 models on hardware that lacks FP4 support?

A new open-source framework called QATFactory lets researchers run quantization-aware training for FP4 formats on GPUs that do not have native FP4 tensor cores. It simulates the deployment-time quantization while keeping matrix multiplications in BF16, then exports checkpoints straight to vLLM and llama.cpp without an extra conversion step.

The problem it solves

Large language model inference is moving toward lower precision to use the throughput of hardware accelerators. Aggressive post-training quantization can degrade model quality. Quantization-aware training normally requires hardware that natively supports the target format. H100 GPUs lack FP4 tensor cores, so teams could not train NVFP4 models directly on them. QATFactory removes that constraint.

The simulation trick

The framework simulates deployment-time quantization while performing matrix multiplications in BF16. Models adapt to quantization noise without needing training hardware that natively supports the target format. For example, it supports NVFP4 training on H100 GPUs. The same approach works for MXFP4 and llama.cpp's Q4_K format. Dense and mixture-of-experts models are both supported. Full-parameter and LoRA-based training are both supported.

Formats and export

Exported checkpoints go directly to vLLM and llama.cpp. No additional lossy conversion step is added. No inference overhead is added. The paper reports experiments on models ranging from 8B to 230B parameters. Evaluated checkpoints run in production inference engines.

Benchmark results

On Qwen3.5-9B, QAD achieves average benchmark accuracies of 68.9% under NVFP4 and 66.0% under MXFP4. The best PTQ results are 65.4% and 56.4% respectively. QAD consistently improves deployed-model quality over strong PTQ baselines across models and formats.

Training strategy depends on format

Although both FP4 formats quantize weights and activations at deployment, the best training strategy is format-dependent. NVFP4 generally performs better when only weights are quantized during training. MXFP4 benefits from quantizing both weights and activations. At a fixed training token budget, training on fewer 32K sequences improves average accuracy by 1.9 points over training on more 4K sequences.

What is released

The complete QATFactory training code and the resulting checkpoints are released. The paper is a preprint submitted on 30 Sep 2026.