Can four-bit n-gram tables cut memory by three quarters without losing performance?

The tables that remember sequences

Many recent open-weight language models add n-gram embedding tables to boost performance. Those tables store learned representations of short token sequences such as pairs or triples of consecutive tokens. The model retrieves them during processing. DeepSeek-V4.1-Flash allocates 196 billion parameters to its Engram n-gram memory. That allocation consumes substantial memory.

Why the memory bill matters

N-gram tables sit alongside the main model weights. They act like a lookup cache for frequent token patterns. When the model sees a familiar pair or triple, it pulls the precomputed representation instead of recomputing it from scratch. This speeds up inference and lifts quality. But the tables grow fast. A 196-billion-parameter table at 16-bit precision needs substantial memory. Even at 8-bit it demands 196 gigabytes. That rules out many deployment scenarios.

Four bits keep almost all the gain

The authors propose storing n-gram embeddings at low precision. They systematically investigate different quantization methods and bit allocations across tables. Across nanochat-, Llama-, and GPT-2-family models trained from scratch, four-bit tables save approximately 75% of table memory. They preserve more than 99% of the validation-loss improvement from adding the tables.

Validation loss stays almost unchanged

Perplexity increases by less than 1.2% on DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next. Table memory drops by 47-55% on those models. Downstream task performance remains largely intact.

Fewer bits hurt more

Using fewer bits leads to a larger drop in performance. A short fine-tuning phase recovers a significant part of this loss. The abstract does not specify how many bits constitute fewer. It also does not quantify the exact recovery rate after fine-tuning.

Bigger tables fit in the same budget

Reinvesting the saved memory in larger tables can improve performance beyond that of the original full-precision tables while still using less memory. This suggests a practical path: compress existing tables, then expand them with the freed capacity. The abstract does not detail the optimal trade-off curve between table size and bit width.

What changes for model builders

Quantization of n-gram tables is now a documented lever. You can shrink the memory footprint dramatically, keep the quality gains, and optionally grow the tables larger than before. The mechanism works on brand-new models trained from scratch and on large existing checkpoints alike.