A preprint on arXiv reports that finetuning a vision-language model on class-imbalanced data can suppress rare-class recall to levels below the model's zero-shot ability. The authors describe this as a regime where standard finetuning actively hurts performance on under-represented classes, rather than merely failing to improve it.

The paper uncovers a "matched-prompt penalty": when the prompt used during finetuning matches the one used at inference, rare-class suppression is more severe. The proposed remedy is cross-prompt inference, where the inference prompt differs from the finetuning prompt. According to the abstract, this approach mitigates the suppression and helps recover rare-class performance.

As a single preprint, the findings have not yet been peer-reviewed or replicated. The abstract does not specify the magnitude of improvement or the exact mechanisms behind the penalty, so those details remain open questions for follow-up work.