arXiv:2604.04974cs.RO2026-04综述被引 3

从视频学机器人操控,无需动作标注

From Video to Control: A Survey of Learning Manipulation Interfaces from Temporal Visual Data

  • 用无标签视频学习机器人控制接口,避免动作标注
  • 提出三类方法:直接策略、隐式中间表示、可解释目标预测
  • 聚焦视频到机器人的衔接难题,适合机器人学习研究者

视频是物理动态的可扩展观测:它捕捉物体运动、接触变化和场景演化过程,无需机器人动作标签。然而将这种时序结构转化为可靠机器人控制仍是开放挑战,因视频缺乏动作监督,且在具身性、视角和物理约束上不同于机器人经验。本综述系统梳理了利用非动作标注时序视频学习机器人操控接口的方法。提出以接口为中心的分类体系,按接口构建位置与控制属性划分,识别出三类方法:直接视频-动作策略(接口隐式)、潜在动作方法(通过紧凑的已学中间表示传递时序结构)、显式视觉接口(预测下游控制的可解释目标)。对每类分析控制集成特性——控制回路如何闭合、执行前可验证内容、失败发生位置。跨类别综合揭示最紧迫的开放挑战集中在机器人集成层:连接视频推断与可靠机器人行为的机制,并据此提出未来研究方向。

原文摘要 · Abstract (English)

Video is a scalable observation of physical dynamics: it captures how objects move, how contact unfolds, and how scenes evolve under interaction -- all without requiring robot action labels. Yet translating this temporal structure into reliable robotic control remains an open challenge, because video lacks action supervision and differs from robot experience in embodiment, viewpoint, and physical constraints. This survey reviews methods that exploit non-action-annotated temporal video to learn control interfaces for robotic manipulation. We introduce an interface-centric taxonomy organized by where the video-to-control interface is constructed and what control properties it enables, identifying three families: direct video-action policies, which keep the interface implicit; latent-action methods, which route temporal structure through a compact learned intermediate; and explicit visual interfaces, which predict interpretable targets for downstream control. For each family, we analyze control-integration properties -- how the loop is closed, what can be verified before execution, and where failures enter. A cross-family synthesis reveals that the most pressing open challenges center on the robotics integration layer -- the mechanisms that connect video-derived predictions to dependable robot behavior -- and we outline research directions toward closing this gap.

机器人操控视频学习控制接口

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。