用视频生成+上下文接口，让机器人跨任务跨设备通用操作
VICX: Generalizable Robot Manipulation via Video Generation and In-Context Operator Network
- 分步执行：先用视频模型生成视觉计划，再由上下文网络转为机械臂动作
- 在Meta-World上实现跨任务、跨机器人的通用操作，无需重新训练
- 仅用图像和状态配对作提示，推理时自动纠错，适合实际部署
通用机器人操作需要在未见场景中进行任务级推理，并可靠地将视觉计划映射到具体执行。为此，我们提出VICX（视频生成与上下文执行），一种解耦的闭环操作框架。其中，冻结的视频生成模型基于视觉语言条件生成高层视觉计划；而视频到轨迹的上下文操作网络（V2T-ICON）作为任务无关接口，将这些计划转化为可执行的机器人状态轨迹。为提升执行泛化能力，V2T-ICON仅使用分割提取的仅机械臂帧观测，并采用检索到的图像-状态对作为上下文提示，实现推理时无需参数更新的鲁棒且泛化的视觉到状态映射。在Meta-World上的实验表明，VICX支持跨任务泛化、闭环自我纠正及跨实体迁移，展现出任务语义与机器人执行双重泛化能力。
原文摘要 · Abstract (English)
Generalizable robot manipulation requires not only task-level reasoning over unseen scenes, but also reliable grounding of visual plans into embodiment-specific execution. To bridge this gap, we propose VICX (Video generation and In-Context eXecution), a decoupled closed-loop manipulation framework. In VICX, a frozen video generation model produces vision-language-conditioned high-level visual plans, while a Video-to-Trajectory In-Context Operator Network (V2T-ICON) serves as the task-agnostic interface that grounds these plans into executable robot-state trajectories. To improve execution generalization, V2T-ICON operates on segmentation-extracted arm-only frame observations and uses retrieved image-state pairs as in-context prompts, allowing a robust and generalizable visual-to-state mapping at inference time without parameter updates. Experiments on Meta-World show that VICX supports cross-task generalization, closed-loop self-correction, and cross-embodiment transfer, demonstrating dual generalization across both task semantics and robot execution. The project webpage can be found here: https://scaling-group.github.io/vicx/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。