How Does Mira Make MoE Models Run Faster on a Single GPU?

The Memory Problem with Mixture-of-Experts Models

Mixture-of-Experts models split their parameters into many small sub-networks called experts. Only a few experts activate for each token. This design lets researchers scale total parameter count without scaling compute per token. The architecture looks attractive for deployment on a single GPU because the active parameter count stays small. The catch is that all expert parameters must still live somewhere. They dominate memory even when inactive. Token-level routing decides which experts fire. That decision is dynamic and unpredictable. It is also skewed. Some experts get used far more often than others. Existing systems react to the router. They wait for the routing output, then move the chosen experts from slow memory into VRAM. This reactive approach wastes cache space. It also prevents overlapping data transfers with computation when VRAM is tight. The GPU sits idle while experts travel across the bus.

How Mira Predicts and Prefetches Experts

Mira changes the stance from reactive to proactive. It adds lightweight per-layer predictors. Each predictor looks at the current layer's routing pattern and forecasts which experts the next two layers will need. The forecast arrives before the router at those layers runs. That lead time lets the system start moving parameters early. The predictors are small enough to add negligible overhead.

They feed a two-tier cache on the GPU. The first tier, called HOT, holds experts that routing telemetry shows are frequently used. The second tier, called STAGE, holds experts the predictors flag as likely upcoming. Token-level routing telemetry manages both tiers. It tracks actual usage to keep HOT accurate. It also validates predictions to avoid cluttering STAGE with mistakes. When a prediction proves wrong, the system evicts the staged expert quickly. This design keeps the cache focused on what the model actually needs.

The Two-Tier Cache and Custom Compression

Moving expert parameters still costs bandwidth. Mira reduces that cost with a custom compression format built for expert tensors. The format cuts metadata size and packs parameters more efficiently than general-purpose codecs. Accuracy loss stays minimal according to the authors. The compressed experts travel between host memory and the GPU faster. The runtime coordinates three moving parts: the predictors, the cache replacement policies, and the quantized transfers. It schedules transfers to overlap with compute kernels wherever possible. Overlap is the key to hiding latency under tight VRAM budgets. The paper describes the runtime as fully integrated. No external orchestrator is required. The system decides when to prefetch, what to evict, and how to pack each transfer without user tuning.

Measured Speedups on Constrained Hardware

The authors benchmark Mira against state-of-the-art baselines on a memory-constrained GPU. Average throughput improves by 5.71x. Time-to-First-Token drops by a factor of 11.71. Beam search inference sees a 3.84x average speedup. These numbers come from experiments listed in the abstract. The paper does not specify which GPU model was used. It does not name the baseline systems. It does not report accuracy metrics for the compressed experts beyond saying degradation is minimal. It does not say how many experts or layers the tested models contain. The submission date is 29 September 2026. The version is v1. The authors are Sanjali Yadav and Bahar Asgari.