研究异构模态下智能体如何发展出有效通信机制。
Learning to Communicate Across Modalities: Perceptual Heterogeneity in Multi-Agent Systems
- 设计异构多步二元通信游戏，模拟不同感知模态的智能体。
- 多模态系统虽需更多信息交换，但能保持分类一致性。
- 通信意义依赖比特分布而非组合结构，适合跨模态迁移研究。
涌现通信为理解智能体如何建立共享结构化表征提供了视角，但现有研究多假设同质模态或对齐的表征空间，忽略了现实场景中的感知异质性。本文研究一种异构多步二元通信任务，其中智能体在模态上存在差异且缺乏感知基础。尽管存在感知错位，多模态系统仍能收敛到与分类一致的消息，并基于感知输入进行表征。单模态系统通信更高效，使用更少比特并实现更低的分类熵；而多模态智能体需更多信息交换，表现出更高不确定性。比特扰动实验表明，意义以分布方式编码，而非组合方式，每个比特的贡献取决于其上下文模式。互操作性分析显示，在不同感知世界中训练的系统无法直接通信，但经有限微调后可实现跨系统通信。本工作将涌现通信定位为研究智能体在异构模态间适应与表征迁移的新框架，为理论与实验开辟新方向。
原文摘要 · Abstract (English)
Emergent communication offers insight into how agents develop shared structured representations, yet most research assumes homogeneous modalities or aligned representational spaces, overlooking the perceptual heterogeneity of real-world settings. We study a heterogeneous multi-step binary communication game where agents differ in modality and lack perceptual grounding. Despite perceptual misalignment, multimodal systems converge to class-consistent messages grounded in perceptual input. Unimodal systems communicate more efficiently, using fewer bits and achieving lower classification entropy, while multimodal agents require greater information exchange and exhibit higher uncertainty. Bit perturbation experiments provide strong evidence that meaning is encoded in a distributional rather than compositional manner, as each bit's contribution depends on its surrounding pattern. Finally, interoperability analyses show that systems trained in different perceptual worlds fail to directly communicate, but limited fine-tuning enables successful cross-system communication. This work positions emergent communication as a framework for studying how agents adapt and transfer representations across heterogeneous modalities, opening new directions for both theory and experimentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。