提出多智能体混合Q学习算法，解决无线网络部分去中心化优化难题。
A Multi-Agent Multi-Environment Mixed Q-Learning for Partially Decentralized Wireless Network Optimization
- 多智能体协作中，通过局部观测和有限信息共享实现联合状态估计。
- 相比集中式算法快50%，平均策略误差仅增加20%；比先进去中心化算法快25%，误差降低40%。
- 适合移动发射机与基站协同的去中心化无线网络，信息开销随节点数线性增长。
Q-learning在无线网络控制与策略优化中表现强大，但面临大状态空间挑战。近期的多环境混合Q-learning（MEMQ）通过在多个相关环境间集成多种Q-learning算法，提升了性能并降低了复杂度。然而，现有MEMQ适用于集中式单智能体网络，不适用于去中心化或多智能体场景。为此，本文提出一种新型多智能体MEMQ算法，用于具有多个移动发射机（TXs）和基站（BSs）的部分去中心化无线网络，其中各TX无法获取其他TX的状态与动作。在非协调状态下，各TX独立行动以最小化自身成本；在协调状态下，TX采用贝叶斯方法基于本地观测估计联合状态，并向主控TX共享有限信息以最小化联合成本。信息共享开销随TX数量线性增长，与联合状态-动作空间大小无关。所提方案比集中式MEMQ快50%，平均策略误差（APE）仅增加20%；比若干先进去中心化Q-learning算法快25%，且APE降低40%。算法收敛性亦得到验证。
原文摘要 · Abstract (English)
Q-learning is a powerful tool for network control and policy optimization in wireless networks, but it struggles with large state spaces. Recent advancements, like multi-environment mixed Q-learning (MEMQ), improves performance and reduces complexity by integrating multiple Q-learning algorithms across multiple related environments so-called digital cousins. However, MEMQ is designed for centralized single-agent networks and is not suitable for decentralized or multi-agent networks. To address this challenge, we propose a novel multi-agent MEMQ algorithm for partially decentralized wireless networks with multiple mobile transmitters (TXs) and base stations (BSs), where TXs do not have access to each other's states and actions. In uncoordinated states, TXs act independently to minimize their individual costs. In coordinated states, TXs use a Bayesian approach to estimate the joint state based on local observations and share limited information with leader TX to minimize joint cost. The cost of information sharing scales linearly with the number of TXs and is independent of the joint state-action space size. The proposed scheme is 50% faster than centralized MEMQ with only a 20% increase in average policy error (APE) and is 25% faster than several advanced decentralized Q-learning algorithms with 40% less APE. The convergence of the algorithm is also demonstrated.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。