让智能体通信更有层次,提升协作效率并大幅减少消息量
HiComm: Hierarchical Communication for Multi-agent Reinforcement Learning
- 通信基于观察的层级结构,分三步选组、选发信者、选具体实体
- 在多个任务中表现优于或持平现有方法,消息量减少最多23倍
- 适合需要高效协作与结构化信息传递的多智能体场景
协同多智能体强化学习常依赖通信缓解观测不完整问题,但现有协议将消息视为无结构的稠密向量,忽略了许多协作环境中观测天然存在的层级结构(如群体与个体)。我们提出HiComm,一种可插拔的通信模块,使消息基于发送方的层级化观测。HiComm为接收方驱动:接收方发出查询后,通过三阶段解码过程依次选择群体、发送方、群体内个体,返回对应特征片段作为消息。该机制将通信从无结构向量传输转变为对发送方观测层级的结构化信息检索。采用可微离散选择的直通Gumbel-Softmax及轻量共享投影设计,可无缝集成至标准MARL流程。在具有不同观测结构和协调需求的任务上实验表明,HiComm性能达到或超越主流学习型通信基线,同时每轮每个接收方通信量最多降低23倍。
原文摘要 · Abstract (English)
Cooperative multi-agent reinforcement learning (MARL) often relies on communication to mitigate partial observability, yet most existing protocols treat messages as flat dense vectors detached from the structure of the observations they summarize. This design overlooks an important source of inductive bias in many cooperative environments, where observations naturally follow a hierarchy such as groups and entities. We propose \textsc{HiComm}, a plug-in communication module that grounds messages in the sender's hierarchical observation. \textsc{HiComm} is receiver-driven: the receiver issues a query, and the hierarchy is resolved through a three-stage decoding process that first selects a group, then a sender, and then an entity within that group, returning the corresponding feature slice as the message. This converts communication from unstructured vector transmission into structured information retrieval over the sender's observation hierarchy. We instantiate this mechanism with Straight-Through Gumbel-Softmax for differentiable discrete selection and a lightweight shared projection design that attaches to standard MARL pipelines. Experiments across cooperative MARL tasks with different observation structures and coordination demands show that \textsc{HiComm} matches or outperforms representative learned communication baselines while reducing communication volume by up to $23\times$ per receiver per episode.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。