Standard text evaluation often reduces a generation system's performance to a single number. That score can rank systems against each other, but it does not explain why one system underperforms or where its outputs go wrong. The authors of a new arXiv preprint argue that this lack of detail makes it hard to act on evaluation results.
They propose 'rate-resolved text evaluation,' a method that uses masked diffusion language models as a diagnostic tool. Instead of collapsing performance into one score, the method varies the masking rate—the fraction of tokens the model is asked to reconstruct—and treats that rate as an axis along which errors can be examined. This allows evaluators to see how generation quality changes as more or less context is provided.
The abstract is brief, so the preprint does not yet describe experimental results or specific use cases. Still, the idea offers a way to turn a model's internal operation into a fine-grained evaluation signal, potentially helping researchers identify whether failures stem from short-range or long-range dependencies, or from other aspects of the generation process.