用可微模型降噪，让千级智能体协作学习更快更稳
Descent-Guided Policy Gradient for Scalable Cooperative Multi-Agent Learning
- 引入可微分析模型生成无噪声梯度信号，修正传统策略梯度的不稳定性
- 理论证明方差从O(N)降至O(1)，样本复杂度与智能体数无关
- 在1500个异构云资源调度智能体任务中20轮内收敛，显著优于基线
规模化协作多智能体强化学习受跨智能体噪声制约。当智能体共享奖励时，每个智能体的学习信号来自依赖所有智能体的共享回报，其他智能体的随机性以跨智能体噪声形式进入信号，且随智能体数N增长。然而，许多工程系统（如云计算、电力系统）具有可微的解析模型，能给出高效系统状态，提供超越噪声共享回报的新参考。本文提出下降引导策略梯度（DG-PG），在策略梯度更新中加入源自可微解析模型的无噪声下降信号。我们证明DG-PG将策略梯度估计器方差从O(N)降至O(1)，保持合作博弈均衡，并实现与智能体数无关的样本复杂度˜O(1/ε)。在最多1500个智能体的异构云资源调度任务中，DG-PG平均在20个回合内收敛，而MAPPO和IPPO在相同架构下无法收敛。
原文摘要 · Abstract (English)
Scaling cooperative multi-agent reinforcement learning (MARL) is fundamentally limited by cross-agent noise. When agents share a common reward, each agent's learning signal is computed from a shared return that depends on all agents, so the stochasticity of the other agents enters the signal as cross-agent noise that grows with $N$. Fortunately, many engineering systems, such as cloud computing and power systems, have differentiable analytical models that prescribe efficient system states, providing a new reference beyond noisy shared returns. In this work, we propose Descent-Guided Policy Gradient (DG-PG), a framework that augments policy-gradient updates with a noise-free descent signal derived from differentiable analytical models. We prove that DG-PG reduces policy-gradient estimator variance from $\mathcal{O}(N)$ to $\mathcal{O}(1)$, preserves the equilibria of the cooperative game, and achieves agent-independent sample complexity $\widetilde{\mathcal{O}} (1/ε)$. On a heterogeneous cloud resource scheduling task with up to 1500 agents, DG-PG converges within 20 episodes on average, while MAPPO and IPPO fail to converge under identical architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。