Mixture-of-experts (MoE) models activate only a few experts per token, which keeps compute low. But when expert weights are offloaded to host memory, the host-to-device (H2D) transfers of those weights enter the critical path of decoding. That transfer cost can dominate latency, especially on memory-constrained devices.

The paper introduces SpecFit, a method that rethinks speculative decoding in this offloaded setting. Speculative decoding normally generates multiple draft tokens and verifies them in parallel, but not all draft tokens require the same expert transfers. SpecFit exploits this asymmetry by accounting for the varying cost of draft tokens, rather than treating them uniformly.

The work is described in arXiv:2610.15417. The abstract focuses on the problem formulation and the proposed approach; no experimental results are included in the available text, so the claims are limited to the method's design rationale. The key insight is that reducing H2D transfer overhead requires looking at which draft tokens actually trigger expensive expert fetches.