改进多语言知识测试数据的语法正确性，提升大模型评估结果的可解释性。
Measuring the Effect of Disfluency in Multilingual Knowledge Probing Benchmarks
- 用谷歌翻译和ChatGPT对斯拉夫语种句子级翻译，修复原模板的语法错误
- 翻译后知识检索得分显著提升，最大增幅达18.7个百分点
- 适用于需高精度多语言评估的研究者，尤其关注语法质量的场景
在多语言大模型事实知识评估中，如MLAMA基准采用模板翻译方式，未考虑插入命名实体后的语法与语义一致性，导致大量语句不合法或措辞错误，尤其对形态丰富的语言影响显著。本文选取MLAMA数据集中4种斯拉夫语种，对比原始模板数据与经谷歌翻译及ChatGPT进行句级翻译后的版本，发现知识检索得分明显提升，并进行定性分析其原因。进一步对5种不同语系的语言进行类似分析，亦观察到相似趋势。因此建议社区在构建高多语言数据集时，通过神经机器翻译或大模型系统实现整句翻译，以保证语法正确性，获得更可靠、可解释的结果。相关数据与代码已开源至GitHub：https://github.com/ZurichNLP/Fluent-mLAMA。
原文摘要 · Abstract (English)
For multilingual factual knowledge assessment of LLMs, benchmarks such as MLAMA use template translations that do not take into account the grammatical and semantic information of the named entities inserted in the sentence. This leads to numerous instances of ungrammaticality or wrong wording of the final prompts, which complicates the interpretation of scores, especially for languages that have a rich morphological inventory. In this work, we sample 4 Slavic languages from the MLAMA dataset and compare the knowledge retrieval scores between the initial (templated) MLAMA dataset and its sentence-level translations made by Google Translate and ChatGPT. We observe a significant increase in knowledge retrieval scores, and provide a qualitative analysis for possible reasons behind it. We also make an additional analysis of 5 more languages from different families and see similar patterns. Therefore, we encourage the community to control the grammaticality of highly multilingual datasets for higher and more interpretable results, which is well approximated by whole sentence translation with neural MT or LLM systems. The dataset and all related code is published at the Github repository: https://github.com/ZurichNLP/Fluent-mLAMA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。