构建英-约鲁巴双语评估数据集，检验大模型跨语言理解能力。
Y-NQ: English-Yorùbá Evaluation dataset for Open-Book Reading Comprehension and Text Generation
- 构建358个问题的英-约鲁巴双语问答数据集，含338篇英文与208篇约鲁巴文档。
- 约鲁巴文本平均仅430词，但模型表现比英文差2.5倍，长文档（>1500词）时性能骤降。
- 揭示当前大模型在低资源语言上的局限性，适合关注多语言AI公平性的研究者。
本文发布一个英-约鲁巴双语开放书阅读理解与文本生成评估数据集，用于评估模型在高资源与低资源语言中的表现。数据集包含358个问题与答案，覆盖338篇英文文档（平均约10,000词）和208篇约鲁巴文档（平均430词）。实验显示，尽管约鲁巴文档更短，其模型表现仍显著落后于英文；在长度相近的少量文档上，约鲁巴性能下降2.5倍。当文档长度达1500词时，约鲁巴性能急剧下降，而英文表现基本稳定。该数据集揭示了当前大模型在跨语言理解上的不平衡，表明其英文能力无法有效迁移至约鲁巴。
原文摘要 · Abstract (English)
The purpose of this work is to share an English-Yorùbá evaluation dataset for open-book reading comprehension and text generation to assess the performance of models both in a high- and a low- resource language. The dataset contains 358 questions and answers on 338 English documents and 208 Yorùbá documents. The average document length is ~ 10k words for English and 430 words for Yorùbá. Experiments show a consistent disparity in performance between the two languages, with Yorùbá falling behind English for automatic metrics even if documents are much shorter for this language. For a small set of documents with comparable length, performance of Yorùbá drops by x2.5 times. When analyzing performance by length, we observe that Yorùbá decreases performance dramatically for documents that reach 1500 words while English performance is barely affected at that length. Our dataset opens the door to showcasing if English LLM reading comprehension capabilities extend to Yorùbá, which for the evaluated LLMs is not the case.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。