Small-to-large language model routing is a cost-saving strategy: a small model handles easy queries, while a larger model is reserved for difficult ones. A new preprint on arXiv argues that the key to making this work is correctly identifying which inputs actually benefit from escalation.
The authors propose a confidence-based mechanism, where low confidence in the small model's output triggers escalation to the larger model. The abstract indicates the method is designed to work across tasks and languages, though it does not specify the models, benchmarks, or datasets used.
Because the paper is a preprint and the abstract is truncated, no numerical results or comparisons with alternative routing strategies are available from the source. The idea is notable for addressing the practical question of when to pay for larger-model inference, rather than assuming all inputs need the same compute.