用简单行为组合实现高精度且通用的复杂操作,无需重新训练
CoStream: Composing Simple Behaviors for Generalizable Complex Manipulation

- 将复杂操作拆解为可组合的独立行为模块
- 在8个真实任务中实现毫米级精度与强泛化能力
- 适合需要高精度装配和抗干扰的机器人应用
长时程、接触密集的复杂操作任务(如将GPU插入PCIe插槽)要求毫米级精度和对新任务的即插即用泛化能力。现有方法难以兼顾:传统流水线依赖特定任务接口,虽精度高但难迁移;端到端模型泛化好,但在分布外任务上缺乏精度,需重新训练。二者均隐含一个假设:掌握的能力必须作为固定流程或整体部署,无法灵活拆分重组。本文提出CoStream框架,通过组合简单独立的行为模块自然涌现复杂操作能力。该框架融合基础模型与多模态感知,生成三种核心行为:基于基础模型的语义行为提取空间约束;通过想象视频追踪关键点预测轨迹;高频触觉与力反馈实现反应式校正。所有输出在共享SE(3)接口上通过右乘合成单一时步位姿指令,由柔顺控制器执行。在8个真实世界任务(涵盖日常操作与精密装配)中验证,尤其在接触密集型装配与物体传输任务中表现最优,并具备执行中手动扰动下的鲁棒恢复能力。
原文摘要 · Abstract (English)
Long-horizon, contact-rich complex manipulation tasks, such as seating a GPU into a PCIe slot, demand both millimeter high precision and out-of-the-box generalization to new tasks. Existing paradigms struggle to satisfy both: classical pipelines use brittle, task-specific interfaces to achieve high-precision control but require costly pipeline redesigns to adapt to new tasks, whereas monolithic end-to-end policies provide better generalization but lack high precision on complex, out-of-distribution tasks unless retrained with new data. Both paradigms share an implicit assumption: once a manipulation capability is acquired, it must be deployed as a rigid pipeline or monolithic whole, rather than being freely decomposed and recomposed. In this paper, we show that complex manipulation capabilities can emerge naturally from the composition of simple, independent behaviors. Rather than deploying a monolithic policy or a rigid pipeline, we propose CoStream, a framework orchestrating foundation models and diverse sensing modalities into multiple composable core behaviors: a semantic behavior extracting spatial constraints via foundation models; a predictive behavior forecasting trajectories by tracking keypoints in imagined videos; and a reactive behavior providing high-frequency tactile and force corrections. On a shared $SE(3)$ interface, these outputs compose by right-multiplication into a single pose command at each control step, executed by a compliant controller. We demonstrate CoStream on 8 real-world tasks spanning everyday manipulation and precision assembly, with the strongest gains in contact-rich assembly and object transfer, and show robust recovery from manual perturbations during execution. Website: https://costream-simple.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。