arXiv:2603.08057cs.ROcs.CV2026-03被引 1

用视觉判断切换机器人技能,让新手轻松教机器人做复杂动作

See and Switch: Vision-Based Branching for Interactive Robot-Skill Programming

  • 将任务建模为带决策节点的技能图,视觉输入决定下一步动作
  • 在79个决策点中实现90.6%的分支选择准确率,异常检测超90%准确
  • 支持手势、力控等交互方式补救执行错误,适合非专家使用

示范编程(PbD)让非专家也能操作机器人,但面对真实世界变化时仍难扩展,尤其在需根据视觉输入在线选择合适任务变体时。本文提出See & Switch框架,将任务表示为由决策状态连接的技能节点图,实现在重放过程中条件分支。其基于眼手视觉的Switcher模块利用实时图像选择下一技能节点,并检测需要新示范的新情况。执行中支持通过力控教学、操纵杆和手势进行恢复示范。在三个精细操作任务上,8名新手用户共收集约900次真实机器人执行数据。为分离视觉决策性能与决策时间误差,我们离线评估Switcher在用户标记的决策窗口内表现。在47个决策状态中,分支选择准确率达90.6%,异常检测准确率超90%,证明了基于视觉输入的可靠条件切换能力。代码与数据已公开于http://imitrob.ciirc.cvut.cz/publications/seeandswitch。

原文摘要 · Abstract (English)

Programming by demonstration (PbD) makes robot programming accessible to non-experts, but scaling it to real-world variability remains a challenge for current teaching frameworks, especially when a robot must select suitable task variants online from visual input. We present See & Switch, an interactive teaching-and-execution framework that represents tasks as graphs of skill parts connected by decision states, enabling conditional branching during replay. Its vision-based Switcher uses eye-in-hand images to select the appropriate successor skill part and detect novel situations that require new demonstrations. The framework supports recovery demonstrations during execution through kinesthetic teaching, joystick control, and hand gestures. We evaluate See & Switch on three dexterous manipulation tasks with 8 novice users, collecting approx. 900 real-robot execution rollouts. To isolate visual decision performance from timing errors during decision states, we evaluate the Switcher offline using user-gated decision state windows. In the evaluation within the decision state windows, the method achieves up to 90.6% branch-selection accuracy and detects anomalies with >90% accuracy in 47 of 79 decision states, demonstrating reliable switching based on visual input for conditional robot-skill programming. We provide all code and experiment data at http://imitrob.ciirc.cvut.cz/publications/seeandswitch.

机器人编程视觉决策示范学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。