提出无需评价网络的序列协作团队信用分配方法,解决多智能体学习中的信号干扰问题。
COSAC: Counterfactual Credit Assignment in Sequential Cooperative Teams
- 用岭回归分解团队奖励,为每个智能体生成独立于同伴噪声的学习信号
- 通过虚构策略延续计算反事实优势,避免环境调用和重要性采样方差
- 在16个智能体的序列任务中表现稳定,适合多智能体协同推理场景
在按固定顺序行动并共享单一团队奖励的协作团队中(如多智能体语言系统、序列机器人任务),每个智能体的信用分配是不确定的。基于评价网络的方法随智能体数量增长而性能下降,因需维护联合或因子化评价网络;现有无评价网络方法存在共用信用导致信号耦合噪声、上游更新过时的重要性采样修正引入指数级方差,或每智能体反事实重播增加环境/奖励调用等缺陷。本文提出COSAC,一种面向序列协作团队的无评价网络个体策略梯度方法。COSAC通过单次岭回归对轨迹批次进行加法式个体奖励分解,使各智能体获得解耦于同伴噪声的学习信号;并通过虚构当前策略延续的前向传播计算反事实优势,替代重要性采样重加权和个体环境重播,且无需额外环境或奖励调用。该估计器实现了我们扩展的序列贵族效用(SeqAU),为Wolpert与Tumer(2001)的贵族效用在序列团队中的推广。我们证明了SeqAU信用的偏差与方差在团队规模增长时仍受控。在序列老虎机的控制实验中,COSAC在最多K=16个智能体的情况下,达到最低的优势均方误差和持续低学习遗憾。在AI2推理挑战(ARC)任务中,四个Qwen3-0.6B智能体轮流推理小学科学问题,COSAC收敛速度优于其他无评价网络基线。
原文摘要 · Abstract (English)
In cooperative teams where agents act in a fixed order and share a single team-level reward (multi-agent language systems, sequential robotic tasks), per-agent credit assignment is under-determined. Critic-based approaches scale poorly as the number of agents grows owing to the costly maintenance of joint/factored critic(s), whereas the existing critic-free alternatives have other issues: common credit across agents that couples every agent's signal to teammate noise, importance-sampling corrections for upstream-update staleness that incur variance exponential in team size, or per-agent counterfactual replay that isolates each agent's effect at the price of extra environment or reward calls. We propose COSAC, a critic-free per-agent policy gradient for sequential cooperative teams. COSAC fits an additive per-agent decomposition of the team reward by a single ridge regression on the rollout batch (giving each agent a learning signal decoupled from teammate noise), and computes each agent's counterfactual advantage from fictitious continuations of the current policy (policy forward passes that replace both importance-sampling reweighting and per-agent environment replay, at no extra environment or reward cost). The estimator instantiates the Sequential Aristocrat Utility (SeqAU), our extension of Wolpert and Tumer's (2001) aristocrat utility to sequential teams. We prove bias and variance bounds on SeqAU credits that stay controlled as the team grows. Our controlled study on sequential bandits demonstrates that COSAC attains the lowest advantage MSE and consistently low learning regret across team sizes up to $K = 16$. On the AI2 Reasoning Challenge (ARC) task, where four Qwen3-0.6B agents reason in turn about a grade-school science question, COSAC attains faster convergence than the other critic-free baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。