How does OPTIC compress per-layer embeddings by 100× while keeping 97.8% of model performance?

Why per-layer embeddings became a compression target

Per-layer embeddings sit inside each transformer layer and grow as models scale. In Gemma 4 E4B they account for 35.2% of all parameters. That share makes them an obvious place to look when memory gets tight. The paper notes that scaling embedding capacity alongside transformer computation is an emerging trend in language models. This trend makes high compression rates increasingly useful.

How OPTIC differs from standard quantization

Conventional quantizers assign every token-layer vector its own tiny code of only a few bytes. OPTIC takes a different route. It stores one quantized code tuple per token and decodes that same tuple at every layer using a layer-specific affine decoder. The codes and decoders are fitted to the PLE output path rather than to the stored vectors themselves. This cross-layer latent vector quantization is what the title describes.

What the benchmarks reveal at three compression levels

At about 100× compression of the BF16 E4B model OPTIC retains an aggregate 97.8% of the uncompressed model's performance across seven benchmarks. Product quantization, the strongest of six conventional baselines, retains 66.3%. At 594× and 1,002× compression OPTIC retains 90.5% and 84.1% respectively. On the official E4B release with 4-bit linear weights and 16-bit activations the method retains 96.9% at about 100× compression.

Memory and speed during inference

During inference at about 100× PLE compression the BF16 E4B model uses 29.7% less GPU memory than with uncompressed PLE at the same key-value cache size. The runtime slowdown is at most 0.40%.

Where the paper stops short

The authors do not report how many E4B models are currently deployed in production. They also do not compare OPTIC against methods that compress the key-value cache itself. Code is available but the exact license terms are not stated in the abstract.