Byte-Pair Encoding (BPE) remains a standard component of modern language models, but the size of its vocabulary is typically decided by a fixed merge budget that is chosen heuristically. A new arXiv paper, "Rethinking BPE: Merge Selection as Model Selection," challenges this practice by suggesting that the choice of merges should be treated as a model selection problem rather than a simple capacity knob.
The authors argue that since larger merge budgets affect downstream model behavior, the selection of merges should be guided by principled criteria—similar to how one selects among competing models—rather than by an arbitrary preset. The abstract notes that tokenizer capacity is "typically determined by a fixed merge budget," and the paper appears to propose a more systematic approach.
While the abstract is brief and does not detail the exact selection method or experimental results, the conceptual shift is clear: tokenizer design deserves the same rigor as model architecture choices. If the approach holds up, it could make tokenizer construction more adaptive and better aligned with the final model's needs, though further details from the full paper would be needed to assess its practical impact.