arXiv:2512.05959cs.CLcs.AI2025-12中稿 · CVPR被引 2

构建多语言多文化多模态检索增强问答基准,揭示大模型在跨语言检索中性能下降问题。

M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAG

  • 构建涵盖42种语言、189个国家的超大规模多模态问答数据集
  • 发现大模型在检索增强下反而性能下降,暴露检索与模型规模不匹配
  • 提示或检索内容非英文时表现显著退化,适合多语言AI研究者参考

视觉语言模型(VLM)在视觉问答(VQA)任务中表现优异,但受限于静态训练数据。检索增强生成(RAG)通过引入实时、文化相关且多语言的信息缓解此问题,然而多语言多模态RAG仍鲜被探索。我们提出M4-RAG,一个覆盖42种语言、56种地区方言与语体、189个国家的超大规模基准,包含超过80,000个具有文化多样性的图像-问题对,用于评估跨语言和跨模态的检索增强型VQA。为兼顾真实性和可复现性,我们构建了包含数百万条精心筛选的多语言文档的受控检索环境,模拟真实检索场景并确保实验一致性。系统评估表明,尽管RAG对小型VLM持续有益,但对大型模型无法有效扩展,甚至导致性能下降,揭示了模型规模与当前检索效率之间的关键不匹配。跨语言评估还发现,当提示或检索上下文使用非英语时,性能显著下降。M4-RAG的代码、数据集及评估协议已开源,地址为https://github.com/davidanugraha/M4-RAG。

原文摘要 · Abstract (English)

Vision-language models (VLMs) have achieved strong performance in visual question answering (VQA), yet they remain constrained by static training data. Retrieval-Augmented Generation (RAG) mitigates this limitation by enabling access to up-to-date, culturally grounded, and multilingual information; however, multilingual multimodal RAG remains largely underexplored. We introduce M4-RAG, a massive-scale benchmark spanning 42 languages, 56 regional dialects and registers, and 189 countries, comprising over 80,000 culturally diverse image-question pairs for evaluating retrieval-augmented VQA across languages and modalities. To balance realism with reproducibility, we build a controlled retrieval environment containing millions of carefully curated multilingual documents relevant to the query domains, approximating real-world retrieval conditions while ensuring consistent experimentation. Our systematic evaluation reveals that although RAG consistently benefits smaller VLMs, it fails to scale to larger models and often even degrades their performance, exposing a critical mismatch between model size and current retrieval effectiveness. Our cross-lingual evaluations also reveal significant performance degradation when prompts or retrieved context are provided in non-English languages. The code, datasets, and evaluation protocols for M4-RAG are available as open-source at https://github.com/davidanugraha/M4-RAG.

多语言多模态RAG视觉问答

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。