Adaptive speculative decoding adjusts the active batch size B and draft length d at every decode step, making the verification width M = B·d a moving target. CUDA Graphs, however, replay a fixed launch configuration, so they cannot directly accommodate this variability. The mismatch forces serving systems to choose between static graphs and adaptive decoding.
PolyGraph, described in a new arXiv paper, introduces plan-buffered CUDA Graphs to bridge this gap. By buffering plans for different configurations, the method aims to retain the low overhead of CUDA Graphs while supporting the dynamic workloads of adaptive speculative decoding. The abstract does not specify the exact buffering mechanism or experimental results, but the approach directly targets the fixed-configuration limitation.