用跨注意力对齐视觉块，提升孟加拉语图像描述的准确性。
Align Where the Words Look: Cross-Attention-Guided Patch Alignment with Contrastive and Transport Regularization for Bengali Captioning
- 通过交叉注意力引导的视觉块对齐，实现真实与合成图像的精准匹配。
- 在Flickr30k-1k和MSCOCO-1k上分别达到BLEU-4 12.29和12.00，显著优于基线。
- 适合低资源语言图像描述任务，尤其适用于孟加拉语等非英语场景。
将视觉-语言模型适配于低资源语言仍具挑战性，因模型常生成关于错误物体的流畅文本。这源于配对数据稀缺、翻译转换导致对齐断裂，以及以英语为中心的预训练忽略目标语言语义。本文提出一种面向孟加拉语的计算感知描述生成流程，基于LaBSE验证的英-孟双语对及11万张双语提示合成图像进行训练。采用冻结的MaxViT提取稳定视觉块，使用孟加拉语原生mBART-50解码器生成文本，并设计轻量级桥梁连接模态。核心创新为三重损失：Patch-Alignment Loss（PAL）利用解码器交叉注意力对齐真实与合成图像块特征；InfoNCE强制真实与合成图像全局分离；基于Sinkhorn的最优传输（OT）确保细粒度块对应平衡。该协同机制提升了定位精度，减少虚假匹配，在Flickr30k-1k上实现BLEU-4 12.29、METEOR 27.98、BERTScore-F1 71.20；在MSCOCO-1k上达到BLEU-4 12.00、METEOR 28.14、BERTScore-F1 75.40，超越强基线并缩小真实-合成中心距41%。
原文摘要 · Abstract (English)
Grounding vision--language models in low-resource languages remains challenging, as they often produce fluent text about the wrong objects. This stems from scarce paired data, translation pivots that break alignment, and English-centric pretraining that ignores target-language semantics. We address this with a compute-aware Bengali captioning pipeline trained on LaBSE-verified EN--BN pairs and 110k bilingual-prompted synthetic images. A frozen MaxViT yields stable visual patches, a Bengali-native mBART-50 decodes, and a lightweight bridge links the modalities. Our core novelty is a tri-loss objective: Patch-Alignment Loss (PAL) aligns real and synthetic patch descriptors using decoder cross-attention, InfoNCE enforces global real--synthetic separation, and Sinkhorn-based OT ensures balanced fine-grained patch correspondence. This PAL+InfoNCE+OT synergy improves grounding, reduces spurious matches, and drives strong gains on Flickr30k-1k (BLEU-4 12.29, METEOR 27.98, BERTScore-F1 71.20) and MSCOCO-1k (BLEU-4 12.00, METEOR 28.14, BERTScore-F1 75.40), outperforming strong CE baselines and narrowing the real--synthetic centroid gap by 41%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。