Speculative decoding that splits work between a mobile device and the cloud can reduce server computation, but it introduces a new bottleneck. Growing a deep draft tree with a small language model (SLM) forces repeated passes on a mobile neural processing unit (NPU), which can be expensive in both time and energy.
The authors of the new paper, HyCo, address this by proposing a hybrid drafting strategy. Rather than relying solely on the mobile NPU to build the entire draft tree, HyCo assigns depth across the mobile and cloud in a way that aims to lower the cost of the repeated passes.
The abstract is brief, but the core idea is clear: balance the drafting workload between device and server to keep the benefits of speculative decoding without overburdening the mobile NPU. Further details on the exact assignment policy are not available in the provided source.