让机器人通过隐式心智模型协作操作,无需直接通信即可高效配合。
Latent Theory of Mind: A Decentralized Diffusion Architecture for Cooperative Manipulation
- 每个机器人维护自身与共有的隐状态,通过理论心智机制推断对方意图。
- 在真实双机械臂实验中,性能超越基础去中心化扩散方法,接近集中式先进模型。
- 适用于多智能体协作场景,尤其适合通信受限的物理机器人系统。
我们提出一种名为隐式心智(LatentToM)的去中心化扩散策略架构,用于协作机器人操作。该策略使多个具有独立感知与计算能力的机械臂能在无显式通信的情况下协同完成共同任务。核心创新在于每个智能体维护两种隐状态:专属的自我嵌入和训练为双方共有的共识嵌入,尽管传感器输入和位姿不同。我们进一步设计解码器,从共识嵌入中推断另一机器人的自我嵌入,类比于潜空间中的心智理论。训练在中心进行,所有策略的共识编码器由受层化理论启发的损失函数监督,引入一阶上同调损失以确保共识嵌入在拓扑流形上的层一致性。为保持共识嵌入表达力,还提出基于心智理论的结构约束与方向性共识机制。执行可完全分布式,无需显式通信,信息通过观察对方动作及其对环境的影响间接传递;也可通过每步共享一次共识嵌入并用层拉普拉斯对齐来利用直接通信。硬件实验表明,LatentToM优于基线去中心化扩散模型,且在双臂操作任务中表现接近最先进的集中式扩散策略。
原文摘要 · Abstract (English)
We present Latent Theory of Mind (LatentToM), a decentralized diffusion policy architecture for collaborative robot manipulation. Our policy allows multiple manipulators with their own perception and computation to collaborate with each other towards a common task goal with or without explicit communication. Our key innovation lies in allowing each agent to maintain two latent representations: an ego embedding specific to the robot, and a consensus embedding trained to be common to both robots, despite their different sensor streams and poses. We further let each robot train a decoder to infer the other robot's ego embedding from their consensus embedding, akin to theory of mind in latent space. Training occurs centrally, with all the policies' consensus encoders supervised by a loss inspired by sheaf theory, a mathematical theory for clustering data on a topological manifold. Specifically, we introduce a first-order cohomology loss to enforce sheaf-consistent alignment of the consensus embeddings. To preserve the expressiveness of the consensus embedding, we further propose structural constraints based on theory of mind and a directional consensus mechanism. Execution can be fully distributed, requiring no explicit communication between policies. In which case, the information is exchanged implicitly through each robot's sensor stream by observing the actions of the other robots and their effects on the scene. Alternatively, execution can leverage direct communication to share the robots' consensus embeddings, where the embeddings are shared once during each inference step and are aligned using the sheaf Laplacian. In our hardware experiments, LatentToM outperforms a naive decentralized diffusion baseline, and shows comparable performance with a state-of-the-art centralized diffusion policy for bi-manual manipulation. Project website: https://stanfordmsl.github.io/LatentToM/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。