构建首个多语言原生视觉语言评估基准，覆盖18种语言
Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation
- 用18种语言的原生题目构建多模态评测集
- 20911道题覆盖14类主题，测试模型跨文化理解能力
- 揭示主流模型在低资源语言中表现差，适合关注公平性研究者
视觉语言模型（VLMs）的评估长期依赖英语基准，导致多语言和跨文化覆盖不足。尽管已有多种多语言基准，但多数基于英文数据集翻译，难以体现文化差异。本文提出Kaleidoscope，目前最全面的多语言视觉语言模型评估基准。该基准为大规模原生多语言多模态评测集，涵盖18种语言与14个主题，共20,911道选择题。通过全球研究人员的开放科学合作构建，确保语言与文化真实性。我们评估了顶尖多语言视觉语言模型，发现其在低资源语言和复杂多模态场景下表现不佳。结果凸显了发展更具文化包容性的多模态评估框架的迫切需求。
原文摘要 · Abstract (English)
The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While multilingual benchmarks have expanded, both in size and languages, many rely on translations of English datasets, failing to capture cultural nuances. In this work, we propose Kaleidoscope, as the most comprehensive exam benchmark to date for the multilingual evaluation of vision-language models. Kaleidoscope is a large-scale, in-language multimodal benchmark designed to evaluate VLMs across diverse languages and visual inputs. Kaleidoscope covers 18 languages and 14 different subjects, amounting to a total of 20,911 multiple-choice questions. Built through an open science collaboration with a diverse group of researchers worldwide, Kaleidoscope ensures linguistic and cultural authenticity. We evaluate top-performing multilingual vision-language models and find that they perform poorly on low-resource languages and in complex multimodal scenarios. Our results highlight the need for progress on culturally inclusive multimodal evaluation frameworks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。