让模型通过翻译任务学会跨语言图像描述,零样本支持新语言。
Scaling Laws for Conditional Emergence of Multilingual Image Captioning via Generalization from Translation
- 用语言前缀引导,让模型从翻译中学到跨语言图像描述能力。
- 在0.4B到11.2B参数模型上验证,性能随模型规模和训练数据量呈可预测增长。
- 适合做多语言视觉任务的通用模型研发者,尤其是资源稀缺语言场景。
跨语言、跨任务迁移受任务特定数据稀缺制约,尤其在语言数量增加时更严重,对视觉语言模型(VLMs)构成挑战。本文研究基于编码器-解码器架构的Transformer型VLM在多语言泛化中的表现,旨在实现仅在翻译任务中出现过的语言的零样本图像描述生成。在此设定下,编码器需学习通用且任务感知的视觉表征,通过插入的交叉注意力层指导解码器。为分析缩放行为,我们在合成数据集上训练了基于Florence-2和Gemma-2的模型(参数量0.4B至11.2B),使用不同计算预算。尽管所有语言均有图像对齐的翻译数据,但仅部分包含图像描述。关键发现:即使某语言仅出现在翻译任务中,也能通过语言前缀实现图像描述能力的涌现。未见任务-语言对的间接学习遵循由模型多语言性、模型规模和已见训练样本共同决定的缩放规律。最后,该缩放规律可推广至下游任务,在多模态机器翻译(Multi30K, CoMMuTE)、词义消歧(CoMMuTE)和图像描述(Multi30K, XM3600, COCO Karpathy)中经微调后达到有竞争力的表现。
原文摘要 · Abstract (English)
Cross-lingual, cross-task transfer is challenged by task-specific data scarcity, which becomes more severe as language support grows and is further amplified in vision-language models (VLMs). We investigate multilingual generalization in encoder-decoder transformer VLMs to enable zero-shot image captioning in languages encountered only in the translation task. In this setting, the encoder must learn to generate generalizable, task-aware latent vision representations to instruct the decoder via inserted cross-attention layers. To analyze scaling behavior, we train Florence-2 based and Gemma-2 based models (0.4B to 11.2B parameters) on a synthetic dataset using varying compute budgets. While all languages in the dataset have image-aligned translations, only a subset of them include image captions. Notably, we show that captioning can emerge using a language prefix, even when this language only appears in the translation task. We find that indirect learning of unseen task-language pairs adheres to scaling laws that are governed by the multilinguality of the model, model size, and seen training samples. Finally, we demonstrate that the scaling laws extend to downstream tasks, achieving competitive performance through fine-tuning in multimodal machine translation (Multi30K, CoMMuTE), lexical disambiguation (CoMMuTE), and image captioning (Multi30K, XM3600, COCO Karpathy).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。