Vision-language models trained on English-centric data often produce fluent captions about the wrong objects when adapted to low-resource languages like Bengali. The authors argue this stems from scarce paired data, translation pivots that break alignment, and pretraining that ignores target-language semantics. Their proposed pipeline addresses this with a compute-aware design: a frozen MaxViT provides stable visual patches, a Bengali-native mBART-50 handles decoding, and a lightweight bridge connects the two modalities.
The core novelty is a tri-loss objective. A Patch-Alignment Loss (PAL) uses decoder cross-attention to align real and synthetic patch descriptors, InfoNCE enforces global separation between real and synthetic features, and Sinkhorn-based optimal transport ensures balanced fine-grained patch correspondence. Together, these losses tighten real-synthetic patch alignment and reduce spurious matches.
On benchmark tests, the method achieves BLEU-4 12.29, METEOR 27.98, and BERTScore-F1 71.20 on Flickr30k-1k, and BLEU-4 12.00, METEOR 28.14, and BERTScore-F1 75.40 on MSCOCO-1k, outperforming strong cross-entropy baselines. The authors also report a 41% reduction in the centroid gap between real and synthetic patches, suggesting the alignment mechanism is directly responsible for the gains. The paper is accepted at PRICAI 2026.