用像素运动统一表征，实现语言驱动的机器人控制
Pixel Motion as Universal Representation for Robot Control
- 以像素运动序列作为中间表征，由文本引导生成
- 在10个任务上实现92%成功率，支持零样本迁移
- 适合需要通用视觉-语言-动作对齐的机器人研究
我们提出LangToMo，一种基于双系统架构的视觉-语言-动作框架，利用像素运动预测作为中间表示。高层系统2（图像扩散模型）从单帧图像生成文本条件的像素运动序列，用于指导机器人控制。像素运动是一种通用、可解释且以运动为中心的表示，可通过弱监督方式从视频中提取，使扩散模型可在任意视频-文本数据上训练。将生成的像素运动视为学习到的通用表征，低层系统1模块通过运动到动作的映射函数将其转化为机器人动作，该函数可手工设计或在少量监督下学习。系统2以稀疏时间间隔运行，作为高层策略；系统1以密集时间间隔运行，作为低层策略。这种分层解耦机制在无监督和有监督设置下均实现灵活、可扩展且泛化性强的机器人控制，弥合了语言、运动与动作之间的鸿沟。
原文摘要 · Abstract (English)
We present LangToMo, a vision-language-action framework structured as a dual-system architecture that uses pixel motion forecasts as intermediate representations. Our high-level System 2, an image diffusion model, generates text-conditioned pixel motion sequences from a single frame to guide robot control. Pixel motion-a universal, interpretable, and motion-centric representation-can be extracted from videos in a weakly-supervised manner, enabling diffusion model training on any video-caption data. Treating generated pixel motion as learned universal representations, our low level System 1 module translates these into robot actions via motion-to-action mapping functions, which can be either hand-crafted or learned with minimal supervision. System 2 operates as a high-level policy applied at sparse temporal intervals, while System 1 acts as a low-level policy at dense temporal intervals. This hierarchical decoupling enables flexible, scalable, and generalizable robot control under both unsupervised and supervised settings, bridging the gap between language, motion, and action. Checkout https://kahnchana.github.io/LangToMo
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。