研究大模型能否跨语言推理，发现其常跳步答题，提出新提示法提升准确率。
Do Language Models Reason Across Languages?
- 设计双跳问答任务，测试模型跨语言推理能力
- 33%情况下模型跳过中间推理仍答对，说明非逐步思考
- 提出三阶段子问题提示法，准确率从10.1%提至66.5%
现实世界信息源天然多语言，这引出了语言模型是否能跨语言整合信息的问题。本文提出一种简单的两跳问答设置，回答问题需基于两份多语言文档进行推理。结果发现，模型在答案片段文档中对语言差异更敏感，尽管两类文档对答题同等重要。在分步子问题评估中，高达33%的多语言案例中，模型虽未完成第一步桥梁信息推理，却仍正确回答了最终问题，表明模型推理并非忠实的分步分解。随后发现，缺乏推理分解导致约18%的组合失败，即两个子问题均答对但最终两跳问题失败。为缓解此问题，本文提出一种三阶段SUBQ提示方法，通过子问题引导多步推理，使准确率从10.1%提升至66.5%。
原文摘要 · Abstract (English)
The real-world information sources are inherently multilingual, which naturally raises a question about whether language models can synthesize information across languages. In this paper, we introduce a simple two-hop question answering setting, where answering a question requires making inferences over two multilingual documents. We find that language models are more sensitive to language variation in answer-span documents than in those providing bridging information, despite the equal importance of both documents for answering a question. Under a step-by-step sub-question evaluation, we further show that in up to 33% of multilingual cases, models fail to infer the bridging information in the first step yet still answer the overall question correctly. This indicates that reasoning in language models, especially in multilingual settings, does not follow a faithful step-by-step decomposition. Subsequently, we show that the absence of reasoning decomposition leads to around 18% composition failure, where both sub-questions are answered correctly but fail for the final two-hop questions. To mitigate this, we propose a simple three-stage SUBQ prompting method to guide the multi-step reasoning with sub-questions, which boosts accuracy from 10.1% to 66.5%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。