用自动生成的双语题对，发现大模型跨语言能力短板。
Cross-Lingual Pitfalls: Automatic Probing Cross-Lingual Weakness of Multilingual Large Language Models
- 用 beam search 和大模型模拟生成双语问题对，探测跨语言差异。
- 在16种语言上构建超6000对数据，揭示多数模型目标语言准确率下降超50%。
- 适合关注多语言模型缺陷与改进的研究者和开发者。
大规模语言模型在自然语言处理中取得显著进展，但其跨语言性能的一致性仍是重大挑战。本文提出一种新方法，高效识别多语言大模型的固有跨语言弱点。该方法利用 beam search 和大模型生成双语问题对，暴露英语与目标语言间的性能差距。我们基于此方法构建了一个包含超过6000对双语样本的数据集，覆盖16种语言，验证了其在揭示先进模型缺陷方面的有效性。大量实验表明，该方法能精确且低成本地定位跨语言弱点，在多种模型中均持续发现目标语言准确率下降超过50%。此外，进一步实验分析了语言相似性与跨语言弱点的关系，发现语言相关性强的语种表现出相似性能模式，并可通过针对性后训练获益。代码已公开于 https://github.com/xzx34/Cross-Lingual-Pitfalls。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved remarkable success in Natural Language Processing (NLP), yet their cross-lingual performance consistency remains a significant challenge. This paper introduces a novel methodology for efficiently identifying inherent cross-lingual weaknesses in LLMs. Our approach leverages beam search and LLM-based simulation to generate bilingual question pairs that expose performance discrepancies between English and target languages. We construct a new dataset of over 6,000 bilingual pairs across 16 languages using this methodology, demonstrating its effectiveness in revealing weaknesses even in state-of-the-art models. The extensive experiments demonstrate that our method precisely and cost-effectively pinpoints cross-lingual weaknesses, consistently revealing over 50\% accuracy drops in target languages across a wide range of models. Moreover, further experiments investigate the relationship between linguistic similarity and cross-lingual weaknesses, revealing that linguistically related languages share similar performance patterns and benefit from targeted post-training. Code is available at https://github.com/xzx34/Cross-Lingual-Pitfalls.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。