arXiv:2505.07787cs.CL2025-05被引 18

让推理模型互相学习,提升纠错能力。

Learning from Peers in Reasoning Models

  • 通过同伴共享中间推理过程,实现动态协作优化
  • 在多个数学基准上平均提升3.3分,最高超基线5分
  • 小模型经微调后性能媲美大模型,适合资源有限场景

大型推理模型(LRMs)具备自我修正能力,但研究发现:若推理起点不佳,模型难以挽回。这一现象称为“前缀主导陷阱”。受心理学中同伴互动促进自纠的启发,我们提出Learning from Peers(LeaP)机制。每个推理路径在每一步生成中间总结,并通过路由机制与其他路径共享,实现推理过程中的同伴协同。针对小模型在摘要与反思指令上表现不佳的问题,我们进一步微调得到LeaP-T系列模型。在AIME 2024、AIME 2025、AIMO 2025和GPQA Diamond上的实验表明,LeaP带来显著提升:例如,QwQ-32B使用LeaP后平均比基线高出近5个百分点;在三个数学基准上超越DeepSeek-R1-671B,平均提升3.3分。值得注意的是,微调后的LeaP-T-7B在AIME 2024上达到DeepSeek-R1-Distill-Qwen-14B水平。深入分析显示,LeaP通过及时引入同伴见解,展现出强鲁棒性与错误容忍度,能应对不同难度任务。该工作标志着大型推理模型在推理过程中实现协同协作的重要进展。代码、数据集及模型已开源。

原文摘要 · Abstract (English)

Large Reasoning Models (LRMs) have the ability to self-correct even when they make mistakes in their reasoning paths. However, our study reveals that when the reasoning process starts with a short but poor beginning, it becomes difficult for the model to recover. We refer to this phenomenon as the "Prefix Dominance Trap". Inspired by psychological findings that peer interaction can promote self-correction without negatively impacting already accurate individuals, we propose **Learning from Peers** (LeaP) to address this phenomenon. Specifically, every tokens, each reasoning path summarizes its intermediate reasoning and shares it with others through a routing mechanism, enabling paths to incorporate peer insights during inference. However, we observe that smaller models sometimes fail to follow summarization and reflection instructions effectively. To address this, we fine-tune them into our **LeaP-T** model series. Experiments on AIME 2024, AIME 2025, AIMO 2025, and GPQA Diamond show that LeaP provides substantial improvements. For instance, QwQ-32B with LeaP achieves nearly 5 absolute points higher than the baseline on average, and surpasses DeepSeek-R1-671B on three math benchmarks with an average gain of 3.3 points. Notably, our fine-tuned LeaP-T-7B matches the performance of DeepSeek-R1-Distill-Qwen-14B on AIME 2024. In-depth analysis reveals LeaP's robust error correction by timely peer insights, showing strong error tolerance and handling varied task difficulty. LeaP marks a milestone by enabling LRMs to collaborate during reasoning. Our code, datasets, and models are available at https://learning-from-peers.github.io/ .

推理模型协作推理自我修正小模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。