用视频预训练知识直接预测机器人动作，无需额外动作训练。
Physical Autoregressive Model for Robotic Manipulation without Action Pretraining
- 用物理标记联合建模视觉与动作，捕捉机器人与环境的动态演化。
- 在PushCube任务上实现100%成功率，动作轨迹与视频预测高度一致。
- 适合想跳过动作预训练、追求高效高精度控制的研究者。
由于操作数据稀缺，现有方法常借助其他模态的预训练大模型。本文提出物理自回归模型（PAR），将帧和动作结合为物理标记，表示机器人与环境的联合演化过程。PAR利用视频预训练中蕴含的世界知识，无需动作预训练即可理解物理动态，实现精准视频预测和一致的动作轨迹。模型采用基于DiT的解标记器，将帧与动作视为连续标记，降低量化误差并促进相互增强。此外，通过引入逆运动学约束、并行训练及KV缓存机制，进一步提升性能与效率。在ManiSkill基准测试中，PAR在PushCube任务上达到100%成功，其他任务表现媲美动作预训练基线，并能准确预测未来视频且动作轨迹紧密对齐。结果表明，从自回归视频预训练迁移世界知识是机器人操作的一条有前景方向。
原文摘要 · Abstract (English)
The scarcity of manipulation data has motivated the use of pretrained large models from other modalities in robotics. In this work, we build upon autoregressive video generation models to propose a Physical Autoregressive Model (PAR), where physical tokens combine frames and actions to represent the joint evolution of the robot and its environment. PAR leverages the world knowledge embedded in video pretraining to understand physical dynamics without requiring action pretraining, enabling accurate video prediction and consistent action trajectories. It also adopts a DiT-based de-tokenizer to model frames and actions as continuous tokens, mitigating quantization errors and facilitating mutual enhancement. Furthermore, we incorporate a causal mask with inverse kinematics, parallel training, and the KV-cache mechanism to further improve performance and efficiency. Experiments on the ManiSkill benchmark show that PAR achieves a 100\% success rate on the PushCube task, matches the performance of action-pretrained baselines on other tasks, and accurately predicts future videos with tightly aligned action trajectories. These findings underscore a promising direction for robotic manipulation by transferring world knowledge from autoregressive video pretraining. The project page is here: https://hcplab-sysu.github.io/PhysicalAutoregressiveModel/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。