让大模型通过自我辩论学习，提升独立解题与协作讨论能力。
Learning from Self-Debate: Preparing Reasoning Models for Multi-Agent Debate
- 训练模型先生成多个解题思路，再基于这些思路进行自我辩论。
- 在多智能体辩论中，性能超越传统方法，且独立解题能力更强。
- 适合需要推理与协作的复杂任务场景，如数学和逻辑问题。
大语言模型（LLM）的推理能力已通过可验证奖励的强化学习（RLVR）显著提升。然而，现有方法通常让模型孤立解题，未为其参与多智能体辩论（MAD）中的不同推理路径做准备。本文提出自辩论强化学习（SDRL），让模型通过生成多个候选解并构建多样推理路径的辩论上下文来学习。模型在初始响应和辩论条件响应上联合优化，既具备独立解题能力，又可有效处理辩论中的多视角推理。在多个基础模型和推理基准上的实验表明，SDRL在不同辩论协议和代理配置下均持续提升MAD表现，同时增强单模型推理能力。
原文摘要 · Abstract (English)
The reasoning abilities of large language models (LLMs) have been substantially improved by reinforcement learning with verifiable rewards (RLVR). At test time, collaborative reasoning through Multi-Agent Debate (MAD) has emerged as a promising approach for enhancing LLM performance. However, current RLVR methods typically train LLMs to solve problems in isolation, without explicitly preparing them to synthesize and benefit from different rationales that arise during debate. In this work, we propose Self-Debate Reinforcement Learning(SDRL), a training framework where models learn from self-debate, equipping a single LLM with both strong standalone problem-solving ability and the capability to process diverse reasoning trajectories in MAD. Given a prompt, SDRL first samples multiple candidate solutions, then constructs a debate context with diverse reasoning paths and generates second-turn responses conditioned on this context. Finally, SDRL jointly optimizes both the initial and debate-conditioned responses, yielding a model that is effective as both a standalone solver and a debate participant. Experiments across multiple base models and reasoning benchmarks show that SDRL consistently improves MAD performance across diverse debate protocols and agent configurations, while simultaneously strengthening single-model reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。