How does OrBIT compress embedding tables 37.9 times by learning their own geometry?

Embedding tables sit at the bottom of every modern language model. They map each token to a vector, and those vectors can occupy more memory than the rest of the network combined.

Compression usually picks a geometry — coordinate blocks, low-rank subspaces, or unrestricted codebooks — and then optimizes inside that choice. OrBIT asks whether the geometry can be learned instead.

Orbit dynamics replace hand-crafted geometry

Most methods decide the coding geometry before seeing the data. OrBIT does the opposite. It extracts local geometry from orbit dynamics — the way embeddings move under repeated transformations — and builds charts from that structure. Each chart covers a region of the embedding space with its own coordinate system. The charts overlap, so errors in one region can be corrected by neighbors when the pieces are glued back together.

The theory shows three things. Tight-chart geometry controls distortion. The global residual directs sequential allocation of the coding budget. Data-geometry-guided refinement improves the codec. After training, the orbit machinery is compiled away. What remains is a compact decoder where the learned structure governs what is stored, where capacity goes, and how local information assembles globally.

Compression numbers on real tables

The authors tested OrBIT on four LLM embedding tables. On GPT-2 the method reaches 37.9× compression relative to 16-bit storage. On each 7B-parameter table it exceeds 23×. Rate-distortion performance stays competitive with established quantization and low-rank baselines.

What the decoder actually stores

The final decoder holds a small set of shared codewords. The learned structure tells the decoder which codewords belong to which chart and how to blend overlapping regions. No orbit computations run at inference time. The geometry that was discovered during training becomes a static map baked into the decoder weights. This means the compression ratio applies directly to the stored model size without runtime overhead.

The approach shifts the design question from "which fixed geometry works best?" to "how do we let the data reveal its own geometry?" For engineers deploying LLMs on memory-constrained hardware, that shift could mean fitting a model that previously required two GPUs onto one. The paper does not measure inference latency changes after compilation.