提出低通信量的分布式强化学习算法，提升多智能体协作效率。
A Communication-Efficient Decentralized Actor-Critic Algorithm
- 每轮通信前本地多次更新策略与价值函数，减少通信次数。
- 理论证明在马尔可夫采样下，样本复杂度为O(ε⁻³)，通信复杂度O(ε⁻¹τ⁻¹)。
- 适用于通信受限的多智能体协同控制场景，尤其适合大规模系统。
本文研究通信受限的多智能体强化学习问题。提出一种去中心化的演员-评论家框架：每个智能体在与邻居交换信息前，先进行多次本地策略和价值函数更新，其中价值函数由多层神经网络近似。该局部训练策略显著降低通信开销，同时保持网络内协调性。在马尔可夫采样条件下，建立了算法的有限时间收敛性分析：达到ε-精确驻点所需的样本复杂度为O(ε⁻³)，通信复杂度为O(ε⁻¹τ⁻¹)，其中τ为本地训练步数。还揭示了最终误差界对神经网络近似能力的依赖关系。在合作控制任务中的数值实验验证了理论结果。
原文摘要 · Abstract (English)
In this paper, we study the problem of reinforcement learning in multi-agent systems where communication among agents is limited. We develop a decentralized actor-critic learning framework in which each agent performs several local updates of its policy and value function, where the latter is approximated by a multi-layer neural network, before exchanging information with its neighbors. This local training strategy substantially reduces the communication burden while maintaining coordination across the network. We establish finite-time convergence analysis for the algorithm under Markov-sampling. Specifically, to attain the $\varepsilon$-accurate stationary point, the sample complexity is of order $\mathcal{O}(\varepsilon^{-3})$ and the communication complexity is of order $\mathcal{O}(\varepsilon^{-1}τ^{-1})$, where tau denotes the number of local training steps. We also show how the final error bound depends on the neural network's approximation quality. Numerical experiments in a cooperative control setting illustrate and validate the theoretical findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。