Why Do LLM Speed Tricks Fail on Apple Silicon?

A new study tests four popular optimization methods on Macs and finds that most of them slow things down instead of speeding them up. The reason is not the chips themselves but the mismatch between how the tricks were designed and how Apple hardware actually works.

The Mismatch Between Design and Reality

Apple Silicon powers a huge share of local language model inference today. Nearly every optimization published in the last few years was built and measured on server-grade NVIDIA GPUs running batched workloads. The paper submitted to arXiv on 22 October 2026 asks a simple question: which of those tricks survive the move to a Mac? The authors ran the same four optimization families across four Apple machines under a single-request decode workload, adding an RTX 4090 where direct comparison was possible. The workload regime matters as much as the silicon because the execution pattern changes the bottleneck.

KV-Cache Pruning Works Here

KV-cache pruning removes less important key-value entries during generation. On Apple hardware it delivers up to 1.75 times end-to-end throughput. The same technique can reduce throughput on a desktop NVIDIA GPU. The difference comes from memory bandwidth pressure: Apple's unified memory benefits more from a smaller cache footprint than the discrete GPU's separate VRAM does. No other family shows a consistent positive result across the Apple lineup.

Speculative Decoding Fails Everywhere

Speculative decoding uses a small draft model to propose tokens that a larger model verifies. Papers report 2 to 6 times speedups on server GPUs. On every Apple machine tested, all evaluated speculative methods reduce throughput. The single-request regime leaves no batch dimension to hide the draft model's overhead. Verification cost exceeds the savings from accepted tokens. The document does not say whether a different draft model or acceptance threshold would flip the sign, nor does it explore alternative scheduling strategies.

Ported Kernels Transfer Partially

CUDA kernel designs transfer only partially. Public Metal lacks asynchronous copy, and batch-one decode makes several serving-oriented mechanisms inapplicable. Transferable components still yield up to 1.20 times decode throughput. Quantization gains vary by machine.

What Makes a Speedup Portable

A reported speedup is not a portable property of a technique alone. It is a joint function of technique, runtime, hardware contract, and workload regime. The paper was accepted at the NeurIPS 2026 Workshop on On-Device Intelligence and the NeurIPS 2026 Workshop on SLMs for Agentic Systems.