用强化学习动态调控多智能体辩论通信结构,提升推理效率与准确性。
RUMAD: Reinforcement-Unifying Multi-Agent Debate
- 将通信拓扑控制建模为强化学习问题,自适应调整连接权重。
- 在多个基准上降低超80%的令牌开销,同时提升推理准确率。
- 无需任务特定训练即可零样本泛化,适合资源受限场景。
多智能体辩论(MAD)系统利用集体智慧增强推理能力,但现有方法难以兼顾准确性、共识形成与计算效率。静态拓扑缺乏对任务复杂度变化的适应性,而外部LLM协调可能引入特权知识,破坏辩论中立性。本文提出RUMAD(强化统一多智能体辩论),将MAD中的动态通信拓扑控制建模为强化学习(RL)问题。RUMAD采用内容无关的观测机制,捕捉高层辩论动态,避免访问原始推理内容。通过多目标奖励函数建模解的质量、凝聚力与效率。使用PPO训练的控制器动态调整通信图边权重,双阈值机制实现对智能体激活与信息可见性的细粒度控制。在MMLU、GSM8K和GPQA基准上的实验表明,RUMAD实现显著效率提升,令牌成本降低超过80%,同时优于单个LLM模型及多个MAD基线的推理准确率。值得注意的是,仅在MMLU上训练的RUMAD展现出对域外(OOD)任务的强零样本泛化能力,表明所学通信策略捕捉了跨任务的有效协同原则。这些结果确立RUMAD为一种高效且鲁棒的多智能体推理部署方案,适用于实际资源约束场景。
原文摘要 · Abstract (English)
Multi-agent debate (MAD) systems leverage collective intelligence to enhance reasoning capabilities, yet existing approaches struggle to simultaneously optimize accuracy, consensus formation, and computational efficiency. Static topology methods lack adaptability to task complexity variations, while external LLM-based coordination risks introducing privileged knowledge that compromises debate neutrality. This work presents RUMAD (Reinforcement-Unifying Multi-Agent Debate), a novel framework that formulates dynamic communication topology control in MAD as a reinforcement learning (RL) problem. RUMAD employs a content-agnostic observation scheme that captures high-level debate dynamics avoiding access to raw agent reasoning content. RUMAD uses a multi-objective reward to model solution quality, cohesion and efficiency. A PPO-trained controller dynamically adjusts edge weights in the communication graph, while a dual-threshold mechanism enables fine-grained control over both agent activation and information visibility. Experimental evaluation across MMLU, GSM8K, and GPQA benchmarks demonstrates that RUMAD achieves substantial efficiency gains, reducing token costs by over 80\%, while still improving reasoning accuracy compared to single LLM model and multiple MAD baselines. Notably, RUMAD trained exclusively on MMLU exhibits robust zero-shot generalization to out-of-domain (OOD) tasks, indicating that the learned communication strategies capture task-independent principles of effective multi-agent coordination. These results establish RUMAD as a efficient and robust approach for deploying multi-agent reasoning application with practical resource constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。