揭秘大模型跨语言检索能力，发现后训练是关键
Understanding LLMs' Cross-Lingual Context Retrieval: How Good It Is And Where It Comes From
- 分两阶段分析跨语言检索：提问编码靠预训练，答案检索靠后训练
- 开源模型表现媲美GPT-4o，后训练显著提升性能上限
- 大模型需多语言后训练才能发挥跨语言潜力，预训练规模非决定因素
跨语言上下文检索（基于一种语言的查询提取另一种语言的上下文信息）是跨语言对齐的基础，但大语言模型（LLMs）在此任务上的表现与机制尚不清晰。本文评估了超过40个LLMs在12种语言上的跨语言上下文检索能力，以跨语言机器阅读理解（xMRC）为典型场景。结果表明，经过后训练的开源模型具备强于预期的跨语言检索能力，可与GPT-4o等闭源模型比肩；其理论最优性能在后训练后显著提升。机制分析显示，该过程分为两个阶段：问题编码（形成于预训练阶段）与答案检索（形成于后训练阶段）。两阶段的稳定性与xMRC性能正相关，而性能瓶颈出现在第二阶段的最后几层，后训练效果在此处尤为明显。此外，更大规模的预训练无法提升xMRC表现；更大的模型需进一步进行多语言后训练，方能充分释放其跨语言上下文检索潜力。
原文摘要 · Abstract (English)
Cross-lingual context retrieval (extracting contextual information in one language based on requests in another) is a fundamental aspect of cross-lingual alignment, but the performance and mechanism of it for large language models (LLMs) remains unclear. In this paper, we evaluate the cross-lingual context retrieval of over 40 LLMs across 12 languages, using cross-lingual machine reading comprehension (xMRC) as a representative scenario. Our results show that post-trained open LLMs show strong cross-lingual context retrieval ability, comparable to closed-source LLMs such as GPT-4o, and their estimated oracle performances greatly improve after post-training. Our mechanism analysis shows that the cross-lingual context retrieval process can be divided into two main phases: question encoding and answer retrieval, which are formed in pre-training and post-training respectively. The phasing stability correlates with xMRC performance, and the xMRC bottleneck lies at the last model layers in the second phase, where the effect of post-training can be evidently observed. Our results also indicate that larger-scale pretraining cannot improve the xMRC performance. Instead, larger LLMs need further multilingual post-training to fully unlock their cross-lingual context retrieval potential.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。