让多智能体在听从指令时仍保持长期目标一致
Robust Instruction Compliance in Cooperative Multi-Agent Reinforcement Learning
- 在指令切换处修正价值函数，避免干扰长期任务
- 实测在复杂协作环境中既高合规又不丢基础性能
- 适合需要灵活响应指令的多智能体系统研究
真实场景中的多智能体强化学习可能需响应外部自然语言指令，打断正在进行的行为并冲突于长期目标。然而，将奖励与指令绑定会引入根本性失效：贝尔曼更新使不同指令上下文下的价值估计相互耦合，导致指令中断长动作时出现价值不一致。本文提出宏动作价值修正（MAVIC），通过修正指令边界处的输入目标，并恢复当前目标下的延续价值，来修正贝尔曼回溯。不同于奖励塑造，MAVIC直接修改自举目标本身，可在统一策略下实现随机指令切换时的一致价值估计。我们提供理论分析与演员-评论家实现，结果表明，MAVIC在日益复杂的协作多智能体环境中，既实现高指令合规性，又保持原有任务性能。
原文摘要 · Abstract (English)
Multi-agent reinforcement learning (MARL) in real-world use cases may need to adapt to external natural language instructions that interrupt ongoing behavior and conflict with long-horizon objectives. However, conditioning rewards on instructions introduces a fundamental failure mode as Bellman updates couple value estimates across instruction contexts, leading to inconsistent values when instructions interrupt macro-actions. We propose Macro-Action Value Correction for Instruction Compliance (MAVIC), which corrects Bellman backups at instruction boundaries by correcting the incoming instruction objective and restoring the continuation value under the current objective. Unlike reward shaping, MAVIC modifies the bootstrapping target itself, enabling consistent value estimation under stochastic instruction switching within a unified policy. We provide theoretical analysis and an actor-critic implementation, and show that MAVIC achieves high instruction compliance while preserving base task performance in increasingly complex cooperative multi-agent environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。