Model routing—choosing which large language model handles a given request—appears to be a simple optimization problem: match each query to the best model. But a new post from IBM Research, published on Hugging Face, argues that the reality is far messier. The authors point out that "best" depends on multiple, often conflicting criteria: accuracy, speed, price, and even the specific nature of the task. A model that excels at code generation may flounder on creative writing, and a cheap, fast model might be good enough for simple queries but fail on nuanced ones.
The post walks through why static routing rules—like always using the largest model for complex prompts and the smallest for trivial ones—quickly break down. Real-world workloads are heterogeneous, and the boundaries between "simple" and "complex" are fuzzy. Moreover, model performance shifts over time as new versions are released and as user behavior changes, so a routing strategy that works today may be obsolete tomorrow. The authors stress that routing is not a one-time setup but an ongoing process of measurement and adjustment.
Because this is a single source, there is no contrasting viewpoint to compare. The post's central claim—that routing is deceptively hard—stands alone. It serves as a caution for teams building LLM-based products: don't assume a fixed router will save you money or improve quality without constant monitoring. The practical takeaway is that routing demands the same rigor as model training itself, including validation sets, live metrics, and periodic re-evaluation.