A new arXiv paper examines how encoding choice affects boundary prediction in hierarchical byte-level language models. These models group bytes into patches using a learned boundary predictor, and the predictor's view of the next codepoint depends on both the encoding and the lookahead available.

The authors note that in UTF-8, codepoints of many scripts take three bytes, which can complicate the prediction task. The abstract suggests that UTF-16 helps in this context, though the full details are not available in the summary.

The paper positions UTF-16 as a potentially better alternative for multilingual boundary prediction, implying that encoding choice is a significant factor in model design. Further reading of the full paper would be needed to understand the exact mechanism and experimental results.