用视觉触觉结合方法,让机器人在陌生环境精准完成接触密集操作。
Touch begins where vision ends: Generalizable policies for contact-rich manipulation
- 分两阶段:先用视觉语言模型定位目标,再用通用触觉策略执行精细操作。
- 在未见环境中成功率超90%,对干扰物有强鲁棒性。
- 适合需要重复使用低层操作技能的机器人系统研发者。
数据驱动方法在精确操作上表现不佳;模仿学习依赖大量难获取的示范,强化学习则产生脆弱且无法泛化的策略。本文提出视觉触觉局部(ViTaL)策略学习框架,将精细操作任务分解为两个阶段:第一阶段利用视觉语言模型(VLM)进行场景级推理,定位目标物体;第二阶段采用可复用、与场景无关的ViTaL策略,基于自我中心视觉和触觉感知执行接触密集型操作。该方法基于一个观察:尽管场景上下文变化,但底层交互模式在不同任务实例间保持一致。通过在标准环境下一次性训练局部策略,可借助‘定位-执行’策略实现泛化。ViTaL在未见环境中对接触密集任务的成功率约为90%,对干扰物具有鲁棒性。其有效性源于三个关键洞见:(1) 基于分割的基础模型可通过对行为克隆训练出鲁棒的视觉编码器;(2) 这些编码器提升了残差强化学习所学策略的泛化能力;(3) 触觉感知显著提升接触密集任务的表现。消融实验验证了每项洞见,且证明ViTaL能良好集成高阶VLM,实现稳健、可复用的底层技能。结果与视频见https://vitalprecise.github.io。
原文摘要 · Abstract (English)
Data-driven approaches struggle with precise manipulation; imitation learning requires many hard-to-obtain demonstrations, while reinforcement learning yields brittle, non-generalizable policies. We introduce VisuoTactile Local (ViTaL) policy learning, a framework that solves fine-grained manipulation tasks by decomposing them into two phases: a reaching phase, where a vision-language model (VLM) enables scene-level reasoning to localize the object of interest, and a local interaction phase, where a reusable, scene-agnostic ViTaL policy performs contact-rich manipulation using egocentric vision and tactile sensing. This approach is motivated by the observation that while scene context varies, the low-level interaction remains consistent across task instances. By training local policies once in a canonical setting, they can generalize via a localize-then-execute strategy. ViTaL achieves around 90% success on contact-rich tasks in unseen environments and is robust to distractors. ViTaL's effectiveness stems from three key insights: (1) foundation models for segmentation enable training robust visual encoders via behavior cloning; (2) these encoders improve the generalizability of policies learned using residual RL; and (3) tactile sensing significantly boosts performance in contact-rich tasks. Ablation studies validate each of these insights, and we demonstrate that ViTaL integrates well with high-level VLMs, enabling robust, reusable low-level skills. Results and videos are available at https://vitalprecise.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。