让机器人仅凭视觉就感知身体状态,靠多模态训练提升视觉理解力。
Kepler-Encoder-v0.1: Towards a Multimodal Embedding Model for Robots

- 用视觉、本体感觉和力矩融合训练,通过自监督学习构建共享隐空间。
- 在摄像头看不到的位置,视觉隐变量能显著恢复末端执行器状态与受力情况。
- 模型跨机器人通用,且无需额外训练即可检测异常状态和重建图像。
机器人需理解自身状态,但摄像头仅能观测部分身体。单帧中力与接触几乎无迹可寻,原始视觉特征对力的预测相关性(R²)低于0.10。本文提出Kepler-Encoder-v0.1,一种以机器人状态为模态的多模态编码器,融合视觉、本体感觉与力/力矩信息,通过学习查询的交叉注意力层生成统一隐表示,并在LeJEPA/SIGReg目标下进行自监督训练。评估时仅输入视觉信息,引发关键问题:训练中融合状态是否使纯视觉隐表示包含像素之外的信息?在RH20T数据集上,答案是肯定的,尤其在摄像头视角最弱处。在未见场景中,该视觉隐表示能显著优于冻结的ViT特征及计算量相当的纯视觉控制基线,恢复末端执行器状态与力值;尽管单时刻力恢复精度有限,但在电机状态(摄像头可见部分)上与最强视觉基线统计持平,且唯一其隐空间几何结构能准确追踪状态变化。一个不依赖具体机体的编码器覆盖四类机器人,数据量匹配对照实验表明其泛化能力源于机体多样性而非数据量。冻结隐表示直接可用:其跨模态预测误差可作零训练异常状态监测(外范围状态AUROC=0.90,场景错位状态AUROC=0.69),且扩散解码器PixNerd可从隐表示重建图像,验证空间压缩保留了世界状态信息。本报告验证单时刻情形;原生速率的时间融合是下一步。
原文摘要 · Abstract (English)
A robot must understand the state of its own body, but a camera sees only part of it. Force and contact leave almost no trace in a single frame, and raw vision features read force at $R^2$ at or below $0.10$ on every robot we test. We present Kepler-Encoder-v0.1, a robot-first multimodal encoder that treats robot state as a modality and fuses vision, proprioception, and force/torque into a single shared latent with a learned-query cross-attention layer, trained self-supervised by masked cross-modal prediction under the LeJEPA/SIGReg objective. At evaluation only vision enters, which poses a sharp question. Does fusing state into training make the vision-only latent carry anything the pixels do not already contain? On the RH20T corpus the answer is yes, precisely where the camera is weakest. On held-out scenes, the vision-only latent recovers end-effector state, and force in particular, significantly above both raw frozen-ViT features and a compute-matched vision-only control on every sensored robot, though absolute force recovery at a single timestep is modest; on motor state, which the camera largely sees, it is statistically tied with the strongest vision baselines, and it is the only feature whose latent geometry tracks state. A single embodiment-agnostic encoder covers four robots, and a data-matched control shows this breadth reflects embodiment diversity rather than data volume. The frozen latent is directly useful. Its own cross-modal prediction error is a training-free invalid-state monitor (AUROC $0.90$ on out-of-range states, $0.69$ on scene-swapped states), and a diffusion decoder (PixNerd) reconstructs the camera frame from the latent, confirming the spatial compression preserves world-state. This report validates the single-timestep case; native-rate temporal fusion is the next step.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。