arXiv:2601.17454cs.MAcs.AI2026-01

在受限身体条件下,集中式学习反而不如独立学习,因同步锁定导致效率下降。

Embodiment-Induced Coordination Regimes in Tabular Multi-Agent Q-Learning

  • 通过隔离协调结构,对比四种配对的多智能体Q学习方法。
  • 独立学习在所有条件下均表现更优,且差距显著(p=0.00195,Cliff's delta=1.0)。
  • 揭示了集中式学习中的时间同步锁机制,适合研究MARL训练动态者阅读。

中心化价值学习是多智能体强化学习中广泛使用的方法,但其优势常被函数逼近和部分可观测性混淆。本文在完全表格化的8×8捕食者-猎物网格世界中,明确引入速度与耐力约束,分离出协调结构,对比四种独立与中心化Q学习组合在三种运动学条件下的表现,共运行10个随机种子。结果显示,在所有条件下,完全独立学习(IQL-IQL)的平均回合长度更短,捕食者回报更高,显著优于完全中心化配置(CQL-CQL),统计显著性p=0.00195,Cliff's delta=1.0。异构组合(IQL-CQL)出现持续性的协调崩溃,未在40,000回合内收敛。即使在冻结的IQL-IQL对手测试下,最强配置仍未达到均衡状态,表明差异源于学习动态而非最终性能。我们提出‘时间同步锁’机制:共享价值函数耦合所有决策,当一智能体耐力受限时,联合Q值迫使高效伙伴进入次优等待,而独立学习者可保持异步追捕。此问题源于信用分配而非函数逼近,推测类似效应可能存在于深层多智能体强化学习中。中心化协调并非总是有益;在具身约束下,其优势减弱或消失,混合配置甚至劣于单一模式。

原文摘要 · Abstract (English)

Centralized value learning underlies a broad class of multi-agent reinforcement learning methods, but its claimed advantage is typically evaluated in settings that confound coordination structure with function approximation and partial observability. We isolate coordination structure in a fully tabular 8x8 predator-prey gridworld with explicit speed and stamina constraints, comparing all four pairings of Independent and Centralized Q-Learning across three kinematic regimes over 10 seeds. Fully independent learning (IQL-IQL) yields shorter episodes and higher predator returns than the fully centralized configuration (CQL-CQL) in every regime and every seed (Wilcoxon p = 0.00195, Cliff's delta = 1.0). Asymmetric IQL-CQL pairings produce coordination breakdowns that persist across the 40,000 episode training budget rather than resolving as transient instability. A best-response test against frozen IQL-IQL predators shows that even the strongest configuration has not converged to equilibrium at 40k episodes, so between-configuration differences reflect learning dynamics under a shared budget rather than end-state performance. We propose a mechanism we call temporal synchronization lock: a shared value function couples all agent decisions, so when one agent is stamina-limited the joint Q-value forces capable partners into suboptimal waits, while independent learners continue asynchronous pursuit. Because this pathology arises from credit assignment rather than function approximation, we conjecture analogous effects may arise in deep MARL methods that centralize credit during training. Centralized coordination is not uniformly beneficial; its advantage over independent learning shrinks or disappears under embodiment constraints, and mixed centralized-independent pairings can perform worse than either uniform choice.

多智能体强化学习协同机制训练动态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。