Multimodal contrastive learning aligns representations from different modalities, such as text and images, by measuring how similar they are. Standard practice relies on manually specified scoring functions, with cosine similarity being a common choice.
The new paper, posted on arXiv, challenges that assumption. It introduces "prior-fitted and frozen scorers" — scoring functions that are fitted to data and then frozen, rather than being fixed ahead of time. This suggests the similarity metric itself can be learned from the data distribution.
The abstract does not include experimental results, so it is not yet clear how much this approach improves over cosine similarity. Still, the idea points toward more adaptive scoring in multimodal models, where the notion of similarity is not assumed but derived.