用多智能体辩论突破单教师纠错上限,提升长序列任务训练稳定性。
MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate

- 让多个教师通过辩论生成更可靠的监督信号,动态加权各教师贡献。
- 在六组配置中均超越现有方法,长序列任务性能提升2.4%,代码生成提升3.7%。
- 适用于复杂推理与代码生成场景,尤其适合高阶智能体系统训练。
在策略蒸馏(OPD)中,学生模型依赖单一教师的逐令牌监督,但当教师出错时,学生会继承错误,导致性能受限。此外,该方法在智能体任务中尚未充分探索,因多步误差累积易引发训练不稳定。本文提出多智能体辩论驱动的在策略蒸馏(MAD-OPD),将教师重构为一个在学生状态上进行辩论的集体,通过辩论生成涌现式集体智慧,提供令牌级监督,各教师贡献按辩论后置信度加权。为拓展至智能体任务,引入在策略智能体蒸馏(OPAD),加入步骤级采样以缓解多步误差累积问题。同时提出任务自适应分歧原则:智能体任务选用JSD(Jensen-Shannon散度),代码生成任务选用反向KL(Kullback-Leibler散度),并通过理论与实证验证。在六组师生配置(Qwen3和Qwen3.5;1.7B-14B学生,8B-32B教师)及五个智能体与代码基准测试中,MAD-OPD在所有配置中排名第一;在14B+8B→4B设置下,智能体平均提升2.4%,代码平均提升3.7%相比更强的单教师OPD。
原文摘要 · Abstract (English)
On-policy distillation (OPD) trains a student on its own trajectories under token-level teacher supervision, but existing methods are capped by a single-teacher capability ceiling: when the teacher errs, the student inherits the error. OPD also remains largely unexplored in agentic tasks, where per-step errors compound across long trajectories and destabilize training. We propose MAD-OPD (Multi-Agent Debate-driven On-Policy Distillation), which breaks this ceiling by recasting the distillation teacher as a deliberative collective of teachers that debate over the student's on-policy state; the debate produces an emergent collective intelligence that supplies token-level supervision, with each teacher's contribution weighted by its post-debate confidence. To extend OPD to agentic tasks, we also introduce On-Policy Agentic Distillation (OPAD), which adds step-level sampling to stabilize training under multi-step error compounding. We additionally derive a task-adaptive divergence principle, selecting JSD (Jensen-Shannon divergence) for agentic stability and reverse KL (Kullback-Leibler) divergence for code generation, and verify it both theoretically and empirically. Across six teacher-student configurations (Qwen3 and Qwen3.5; 1.7B-14B students, 8B-32B teachers) and five agentic and code benchmarks, MAD-OPD ranks first across all six configurations; on the 14B+8B$\to$4B setting it lifts the agentic average by $+2.4\%$ and the code average by $+3.7\%$ over the stronger single-teacher OPD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。