用分层代理解决大模型长期决策漂移问题
Multi$^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments
- 高阶代理生成目标，低阶代理执行动作，分工明确
- 在多轮交互中表现更稳定，显著减少目标偏离
- 适合需要长期规划的智能体系统研究者
大语言模型研究的核心目标之一是构建能持续与动态环境互动的智能体系统。尽管当前基于LLM的智能体展现出出色的上下文推理能力，其长期决策仍易出现目标漂移问题。本文提出Multi²框架，通过分层设计将代理行为分解为互补角色：高层代理（系统1）利用监督微调生成情境感知的子目标，底层代理（系统2）通过离线到在线的强化学习执行原子动作。这种分离机制实现了稳定的长周期控制，缓解了目标漂移，并支持高效适应。在多种交互环境中，Multi²均优于主流基线，展现出更强的鲁棒性与协作能力。此外，本文还构建并发布三个分层基准数据集，填补了基于LLM的分层决策训练与评估的长期空白。
原文摘要 · Abstract (English)
A central goal of large language model (LLM) research is to build agentic systems that can plan, act, and adapt through sustained interaction with dynamic environments. While recent LLM-based agents exhibit impressive contextual reasoning, their long-horizon decision-making remains fragile, often suffering from objective drift, where goals and plans drift over extended interactions. We introduce Multi$^2$, a hierarchical multi-agent decision-making framework that explicitly decomposes agent behavior into complementary roles. A high-level agent (System 1) focuses on context-aware sub-goal generation using supervised fine-tuning (SFT), while a low-level agent (System 2) executes atomic actions through offline-to-online reinforcement learning (RL) in interactive environments. This separation enables stable long-horizon control, mitigates objective drift, and allows efficient adaptation. Across diverse interactive environments, Multi$^2$ consistently outperforms strong agentic baselines, demonstrating improved robustness and coordination in multi-turn interaction. Beyond performance, we introduce and release three hierarchical benchmark datasets, filling a long-standing gap in training and evaluating hierarchical decision-making for LLM-based agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。