将多智能体辩论蒸馏为单模型，大幅降低推理成本且便于控制有害行为。
Latent Agents: A Post-Training Procedure for Internalized Multi-Agent Debate
- 通过两阶段微调将多智能体辩论过程内化到单一语言模型中
- 在多个基准上用最多93%更少的词元达到相同或更好性能
- 发现内化后存在可解释的智能体特异性激活子空间，利于行为控制
多智能体辩论已被证明能提升大语言模型的推理能力，但其计算开销大，需生成长对话才能回答问题。为此，我们提出一种框架，通过结合辩论结构学习与动态奖励调度、长度截断的内化方法，将多智能体辩论蒸馏为单个大语言模型。在多个模型和基准测试中，内化模型使用最多93%更少的词元即可达到或超过显式多智能体辩论的表现。我们进一步通过激活引导研究其机制，发现内化过程形成了智能体特异性的子空间：对应不同智能体视角的激活空间可解释方向。我们还展示了实际应用：通过内化辩论将恶意智能体植入大语言模型，并施加负向引导以抑制其行为，结果表明，相比直接引导基础模型，该方法在更小性能损失下实现对有害行为的更精准定位与控制。研究为理解蒸馏模型中的多智能体能力提供了新视角，并为控制内化推理行为提供了实用指导。代码已公开于 https://github.com/johnsk95/latent_agents。
原文摘要 · Abstract (English)
Multi-agent debate has been shown to improve reasoning in large language models (LLMs). However, it is compute-intensive, requiring generation of long transcripts before answering questions. To address this inefficiency, we develop a framework that distills multi-agent debate into a single LLM through a two-stage fine-tuning pipeline combining debate structure learning with internalization via dynamic reward scheduling and length clipping. Across multiple models and benchmarks, our internalized models match or exceed explicit multi-agent debate performance using up to 93% fewer tokens. We then investigate the mechanistic basis of this capability through activation steering, finding that internalization creates agent-specific subspaces: interpretable directions in activation space corresponding to different agent perspectives. We further demonstrate a practical application: by instilling malicious agents into the LLM through internalized debate, then applying negative steering to suppress them, we show that distillation makes harmful behaviors easier to localize and control with smaller reductions in general performance compared to steering base models. Our findings offer a new perspective for understanding multi-agent capabilities in distilled models and provide practical guidelines for controlling internalized reasoning behaviors. Code available at https://github.com/johnsk95/latent_agents
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。