Speculative Decoding: Faster LLM Inference and a New Control Strategy

Speculative decoding accelerates autoregressive generation by having a smaller draft model propose several tokens at once, which a larger target model then verifies in parallel. Both sources describe this core mechanism, and both note that it does not alter the target model's output distribution—the speedup comes from parallel verification rather than from changing what the model would have generated.

The sources differ in scope. The Unite.AI guide focuses on the general trade-offs of speculative decoding, such as the balance between draft-model quality and verification overhead, and how to evaluate and control the technique. The arXiv paper, by contrast, targets a specific trade-off between two drafting strategies: neural drafting, where a model generates proposals, and context-based copying, where tokens are reused from the input context. The paper argues that existing approaches force a choice between these strategies and proposes using intrinsic signals from the model itself to decide which to apply.

Taken together, the sources agree on the underlying mechanics but approach them from different angles. The explainer offers a practical overview, while the research paper contributes a more granular control mechanism for improving speculative decoding in settings where both drafting strategies are viable.