让大模型在多语言下也能精准推理，提升复杂问题解决能力。
Demystifying Multilingual Chain-of-Thought in Process Reward Modeling
- 构建跨七种语言的推理评分模型，基于翻译数据训练
- 在11种语言上平均准确率提升，早期推理错误减少
- 适合需要多语言复杂推理的应用，如跨国智能助手
大型语言模型（LLMs）被设计用于执行多种任务。为提升其处理需多步推理的复杂问题的能力，近期研究采用过程奖励建模（PRM）在强化学习中对每一步推理提供细粒度反馈，但主要集中在英语。本文解决将PRM扩展至多语言环境的关键挑战，通过在涵盖七种语言的数据集上训练多语言PRM实现，该数据集源自英文翻译。在两个广泛使用的推理基准上，对11种语言进行综合评估，结果表明多语言PRM不仅提升平均准确率，还减少早期推理错误。此外，研究发现多语言PRM对训练语言数量和英文数据量敏感，并揭示更多候选响应和可训练参数带来的优势。本工作为复杂多步推理任务的稳健多语言应用开辟了新路径。代码已开源，以推动该方向研究。
原文摘要 · Abstract (English)
Large language models (LLMs) are designed to perform a wide range of tasks. To improve their ability to solve complex problems requiring multi-step reasoning, recent research leverages process reward modeling to provide fine-grained feedback at each step of the reasoning process for reinforcement learning (RL), but it predominantly focuses on English. In this paper, we tackle the critical challenge of extending process reward models (PRMs) to multilingual settings. To achieve this, we train multilingual PRMs on a dataset spanning seven languages, which is translated from English. Through comprehensive evaluations on two widely used reasoning benchmarks across 11 languages, we demonstrate that multilingual PRMs not only improve average accuracy but also reduce early-stage reasoning errors. Furthermore, our results highlight the sensitivity of multilingual PRMs to both the number of training languages and the volume of English data, while also uncovering the benefits arising from more candidate responses and trainable parameters. This work opens promising avenues for robust multilingual applications in complex, multi-step reasoning tasks. In addition, we release the code to foster research along this line.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。