The abstract for the UltraBERT paper, posted on arXiv, highlights a disconnect in modern encoder development. It notes that recent bidirectional encoders have been modernized with innovations originally developed for causal large language models. However, the pretraining recipe used for these encoders has not changed in the same way.

The authors argue that this pretraining recipe is still inherited from small-scale conventions. As a result, encoders may be left underoptimized, even as their architectures have advanced. The abstract does not yet describe the specific solution proposed by UltraBERT, but it frames the pretraining method as a key bottleneck.

Because the source is only an abstract, the full details of the proposed approach are not available. The significance of the claim is that it points to pretraining, rather than architecture alone, as a limiting factor for encoder performance.