arXiv:2601.10096cs.LG2026-01Conference of the …

用英语文本训练轻量映射层,让多语言文本直接接入多模态模型。

Multilingual-To-Multimodal (M2M): Unlocking New Languages with Monolingual Text

  • 仅用英语文本训练少量线性层,将多语言嵌入映射到多模态空间。
  • 在11种语言上零样本迁移达89.5%召回率,接近英文基线94.9%。
  • 方法简单高效,适用于图文、音文检索及生成任务,适合多语言研究者。

多模态模型在英语上表现优异,但其他语言因多语言多模态数据稀缺而性能下降。现有方案依赖机器翻译,忽视了多语言文本建模进展。本文提出M2M,一种轻量对齐方法:仅用英语文本训练少量线性层,将多语言文本嵌入映射至多模态空间。尽管结构简单,其在英文上达到94.9% Recall@10,跨11种语言的零样本迁移平均达89.5% Recall@10(10种未见语言)。t-SNE可视化显示多语言嵌入与多模态表示高度对齐,权重分析表明该变换重塑了嵌入几何结构而非简单旋转。M2M还拓展至音频-文本检索和文本-图像生成任务。代码与检查点已开源,并提供MSCOCO Multilingual 30K、AudioCaps Multilingual、Clotho Multilingual等多语言评估数据集。

原文摘要 · Abstract (English)

Multimodal models excel in English, supported by abundant image-text and audio-text data, but performance drops sharply for other languages due to limited multilingual multimodal resources. Existing solutions rely on machine translation, while advances in multilingual text modeling remain underutilized. We introduce M2M, a lightweight alignment method that learns only a few linear layers--using English text alone--to map multilingual text embeddings into multimodal space. Despite its simplicity, M2M matches baseline performance in English (94.9% Recall@10) and achieves strong zero-shot transfer (89.5% Recall@10 averaged across 11 languages, 10 unseen) on XTD Text-to-Image retrieval. Qualitative t-SNE visualizations show that multilingual embeddings align tightly with multimodal representations, while weight analysis reveals that the transformation reshapes embedding geometry rather than performing trivial rotations. Beyond image-text retrieval, M2M demonstrates robustness across datasets and tasks, extending to Audio-Text retrieval and Text-to-Image generation. We release code and checkpoints (https://github.com/piyushsinghpasi/M2M) along with multilingual evaluation datasets: MSCOCO Multilingual 30K (https://huggingface.co/datasets/piyushsinghpasi/mscoco-multilingual-30k), AudioCaps Multilingual (https://huggingface.co/datasets/piyushsinghpasi/audiocaps-multilingual), and Clotho Multilingual (https://huggingface.co/datasets/piyushsinghpasi/clotho-multilingual).

多语言多模态零样本文本映射

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。