用文化知识增强多语言多模态模型，提升对小众文化的理解能力。
Grounding Multilingual Multimodal LLMs With Cultural Knowledge
- 基于维基数据构建文化实体图像与多语言问答数据
- 训练出的CulturalPangea模型在42国39语言上表现领先，平均提升5.0分
- 适合关注跨文化理解与全球包容性AI的研究者
多模态大模型在高资源场景表现优异，但常误读长尾文化实体且在低资源语言中表现欠佳。为此，我们提出一种以数据为中心的方法，直接将多模态大语言模型（MLLMs）锚定于文化知识。利用维基数据的大规模知识图谱，我们收集代表重要文化实体的图像，并生成合成的多语言视觉问答数据。由此构建的数据集CulturalGround包含2200万条高质量、富含文化信息的视觉问答对，覆盖42个国家和39种语言。我们在该数据集上训练开源模型CulturalPangea，同时穿插标准多语言指令微调数据以保持通用能力。CulturalPangea在多个聚焦文化的多语言多模态基准上达到开源模型最优表现，平均性能优于先前模型5.0分，且未降低主流视觉-语言任务表现。研究结果表明，这种针对性的文化锚定方法可显著缩小MLLMs中的文化差距，为构建全球包容的多模态系统提供可行路径。
原文摘要 · Abstract (English)
Multimodal Large Language Models excel in high-resource settings, but often misinterpret long-tail cultural entities and underperform in low-resource languages. To address this gap, we propose a data-centric approach that directly grounds MLLMs in cultural knowledge. Leveraging a large scale knowledge graph from Wikidata, we collect images that represent culturally significant entities, and generate synthetic multilingual visual question answering data. The resulting dataset, CulturalGround, comprises 22 million high-quality, culturally-rich VQA pairs spanning 42 countries and 39 languages. We train an open-source MLLM CulturalPangea on CulturalGround, interleaving standard multilingual instruction-tuning data to preserve general abilities. CulturalPangea achieves state-of-the-art performance among open models on various culture-focused multilingual multimodal benchmarks, outperforming prior models by an average of 5.0 without degrading results on mainstream vision-language tasks. Our findings show that our targeted, culturally grounded approach could substantially narrow the cultural gap in MLLMs and offer a practical path towards globally inclusive multimodal systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。