将多语言大模型与视觉模块结合，实现多模态翻译
Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translation
- 用ViT提取图像特征，通过适配层映射到大模型空间
- 在印地语多模态翻译中达到当前最优，三任务均表现领先
- 适合关注多语言、跨模态翻译的开发者和研究者
本文介绍了我们在2024年亚洲翻译研讨会（WAT2024）英文到低资源多模态翻译任务中的系统方案。我们提出Chitranuvad模型，将多语言大模型与视觉模块有效融合，用于多模态翻译。方法采用ViT图像编码器提取视觉表征作为视觉标记嵌入，通过适配层投影至大模型空间，并以自回归方式生成翻译。我们在印地语、孟加拉语和马拉雅拉姆语三种语言的任务中参与了图像描述、纯文本及多模态翻译三个赛道，对印地语在挑战集上取得当前最优（SOTA）结果，其他语言也保持竞争力。
原文摘要 · Abstract (English)
In this work, we provide the system description of our submission as part of the English to Lowres Multimodal Translation Task at the Workshop on Asian Translation (WAT2024). We introduce Chitranuvad, a multimodal model that effectively integrates Multilingual LLM and a vision module for Multimodal Translation. Our method uses a ViT image encoder to extract visual representations as visual token embeddings which are projected to the LLM space by an adapter layer and generates translation in an autoregressive fashion. We participated in all the three tracks (Image Captioning, Text only and Multimodal translation tasks) for Indic languages (ie. English translation to Hindi, Bengali and Malyalam) and achieved SOTA results for Hindi in all of them on the Challenge set while remaining competitive for the other languages in the shared task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。