用多模态重写框架减少跨语言图像描述偏差。
A Multimodal Recaptioning Framework to Account for Perceptual Diversity Across Languages in Vision-Language Modeling
- 用母语数据+最近邻引导+多模态推理增强目标语言描述。
- 德语和日语图文检索平均召回率提升最高达3.5，原生/翻译错误减少4.4。
- 适合关注跨语言视觉-语言建模与文化差异的研究者。
人们描述图像时会因感知差异使用不同词汇或突出不同细节，尤其在跨语言和跨文化背景下更为显著。当前视觉-语言模型（VLM）通常依赖英文标题的机器翻译进行多语言训练，但输入内容源自英语母语者的感知视角，导致感知偏差。本文提出一种多模态重写框架，利用少量母语数据、最近邻示例引导及多模态大模型推理，生成更符合目标语言实际描述习惯的图像标题。将重写后的标题用于多语言CLIP微调，在德语和日语图文检索任务中，平均召回率最高提升3.5，原生描述与翻译错误对比降低4.4。此外，该方法还支持分析跨语言物体描述差异，并为跨数据集与跨语言泛化提供新见解。
原文摘要 · Abstract (English)
When captioning an image, people describe objects in diverse ways, such as by using different terms and/or including details that are perceptually noteworthy to them. Descriptions can be especially unique across languages and cultures. Modern vision-language models (VLMs) gain understanding of images with text in different languages often through training on machine translations of English captions. However, this process relies on input content written from the perception of English speakers, leading to a perceptual bias. In this work, we outline a framework to address this bias. We specifically use a small amount of native speaker data, nearest-neighbor example guidance, and multimodal LLM reasoning to augment captions to better reflect descriptions in a target language. When adding the resulting rewrites to multilingual CLIP finetuning, we improve on German and Japanese text-image retrieval case studies (up to +3.5 mean recall, +4.4 on native vs. translation errors). We also propose a mechanism to build understanding of object description variation across languages, and offer insights into cross-dataset and cross-language generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。