用对话式语言模型统一处理多智能体离线决策,提升泛化能力
DLM: Unified Decision Language Models for Offline Multi-Agent Sequential Decision Making

- 将多智能体决策建模为对话序列生成,支持异构观测与动作
- 两阶段训练:监督微调+群体相对策略优化,显著提升对分布外动作的鲁棒性
- 零样本泛化能力强,适用于未见过的任务场景,适合复杂决策系统设计
从离线数据集中构建可扩展且可复用的多智能体决策策略仍是离线多智能体强化学习(MARL)的核心挑战,现有方法通常依赖固定观测格式和动作空间,限制了泛化能力。相比之下,大语言模型(LLM)提供了灵活的建模接口,能自然处理异构观测与动作。受此启发,我们提出决策语言模型(DLM),将多智能体决策建模为集中训练、分散执行范式下的对话式序列预测问题。DLM采用两阶段训练:第一阶段为监督微调,利用对话式数据集进行集中训练,结合智能体间上下文,从离线轨迹中生成可执行动作;第二阶段为群体相对策略优化,通过轻量级奖励函数增强对分布外动作的鲁棒性。在多个基准测试上,统一的DLM性能优于强基线方法及基于LLM的对话式决策方法,并展现出对未见场景的强零样本泛化能力。
原文摘要 · Abstract (English)
Building scalable and reusable multi-agent decision policies from offline datasets remains a challenge in offline multi-agent reinforcement learning (MARL), as existing methods often rely on fixed observation formats and action spaces that limit generalization. In contrast, large language models (LLMs) offer a flexible modeling interface that can naturally accommodate heterogeneous observations and actions. Motivated by this, we propose the Decision Language Model (DLM), which formulates multi-agent decision making as a dialogue-style sequence prediction problem under the centralized training with decentralized execution paradigm. DLM is trained in two stages: a supervised fine-tuning phase, which leverages dialogue-style datasets for centralized training with inter-agent context and generates executable actions from offline trajectories, followed by a group relative policy optimization phase to enhance robustness to out-of-distribution actions through lightweight reward functions. Experiments on multiple benchmarks show that a unified DLM outperforms strong offline MARL baselines and LLM-based conversational decision-making methods, while demonstrating strong zero-shot generalization to unseen scenarios across tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。