What makes ternary matrix multiplication faster when most weights are zero?

The problem with ternary weights

Large language models demand heavy compute. Ternary quantization compresses weights to three values — minus one, zero, plus one — and often leaves 50 to 90 percent of them as zero. Existing systems like BitNet and RSR handle the ternary part well but ignore the sparsity pattern. Standard sparse formats do the opposite: they track zeros but lose the ternary structure. Both approaches leave performance on the table.

The gap matters because ternary models already run on edge devices. Every skipped operation saves battery and latency. Yet no current kernel exploits both properties at once. The paper frames this as a dual optimization opportunity that existing work foregoes.

A format built for both properties

SSR introduces a dedicated ternary data layout. It stores only non-zero values and their positions, using a compact encoding that preserves the sign. The format avoids the overhead of general sparse matrices while keeping the ternary arithmetic simple. No separate index arrays. No branching on value type. The layout feeds directly into the next stage.

This design means the memory footprint shrinks with sparsity. At 90 percent zeros the representation approaches the theoretical minimum for ternary data. The authors do not publish exact byte counts per layer.

Computation trees that grow with sparsity

The algorithm organizes multiply-add operations into trees. Each tree merges partial sums from non-zero elements. As sparsity rises, the trees shrink. Fewer leaves mean fewer additions. The depth scales logarithmically with the number of non-zero entries per row. This is not a fixed schedule — it adapts to the actual pattern in each layer.

Tree construction happens once per weight matrix. The cost amortizes across inference runs. For static models this is a one-time overhead. The paper does not discuss dynamic sparsity or fine-tuning scenarios.

Measured gains on GEMM and full models

On ternary GEMM kernels with 45 to 95 percent sparsity, SSR beats RSR++ by 2.1 to 11.3 times. The gap widens as zeros increase. End-to-end inference on Llama-3 1B shows 3.5 to 6.3 times speedup over RSR++. Memory drops by 4.9 percent. The paper reports these figures for the DATE 2026 publication. No results are given for larger models or training workloads.

Speedup numbers come from GPU implementations. CPU and accelerator backends are not evaluated. The authors note that RSR++ itself improves on BitNet and RSR, so the baseline is already optimized for ternary arithmetic.

Where the advantage fades

Below 50 percent sparsity the theoretical edge disappears. The trees no longer shrink enough to offset their management cost. The authors do not specify a hard crossover point. They also do not compare against dense ternary kernels on hardware without sparse support. Deployment on such devices remains an open question.

Real-world sparsity depends on quantization recipe and model size. The 50 to 90 percent range cited in the abstract comes from prior ternary LLM work, not from SSR's own measurements. How often production models fall below the threshold is not addressed.