用射线信息增强视觉模型，让机器人更抗摄像头位置变化。
RayViT: Ray-Conditioned Visual Representations for Viewpoint-Robust Imitation Learning
- 将相机几何信息转为射线图，注入ViT模型提升空间感知。
- 在多任务仿真和真实机器人任务中，鲁棒性提升13个百分点。
- 适合需要稳定视觉导航的机器人仿学习场景。
视觉模仿学习使机器人能直接从图像中习得视觉运动技能，但RGB观测缺乏明确的几何线索，导致学习策略对摄像头扰动敏感。为此，我们提出轻量级架构RayViT（射线条件视觉变换器编码器），将相机几何信息注入预训练ViT骨干网络。RayViT将相机几何表示为普吕克射线图，将其分块为射线特征，并通过门控交叉注意力生成射线条件类标记。这些射线特征作为密集位置嵌入，而射线类标记取代原ViT类标记，提供几何感知的摘要表征。我们结合辅助余弦相似性损失，持续提升几何感知标记的性能与鲁棒性。在模拟和真实机器人任务上的实验表明，RayViT在多任务RoboCasa基准下，摄像头扰动时鲁棒性提升约13个百分点；在真实世界多任务成功率上，平均完成阶段数提升1.78，优于基线方法。
原文摘要 · Abstract (English)
Visual imitation learning enables robots to acquire visuomotor skills directly from images, yet RGB observations lack explicit geometric cues, making learned policies brittle to camera perturbations. To address this, we propose \textbf{Ray-conditioned Vision Transformer Encoder (RayViT)}, a lightweight architecture that injects camera geometry into pretrained ViT backbones. RayViT represents camera geometry as a Plücker ray map, patchifies it into ray features, and uses gated cross-attention to produce a ray-conditioned class token. These ray features are added as dense positional embeddings, while the ray class token replaces the original ViT class token to provide a geometry-aware summary representation. We combine this approach with an auxiliary cosine similarity loss to consistently improve the performance and robustness for geometry-aware tokens. Experiments on sim- and real-robot tasks demonstrate that RayViT improves robustness by approximately 13 percentage points under camera perturbations in multi-task RoboCasa benchmark and by 1.78 average completed stages in real-world multi-task success rate compared to baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。