用密集视觉特征提升机器人控制效率,速度更快、效果更好。
Patch Policy: Efficient Embodied Control via Dense Visual Representations

- 直接使用预训练视觉变换器的密集补丁特征,不依赖全局编码
- 在4个仿真和3个真实环境上比现有方法提升40%性能
- 参数量仅为OpenVLA-OFT的0.7%,适合高频实时控制
基于视觉变换器(ViT)的预训练密集视觉特征在机器人学习中尚未被充分利用。当前机器人策略通常将观测压缩为单一全局标记,或从头训练视觉主干,导致丢失细粒度空间信息并放弃大规模视觉预训练的优势。尽管存在如大型视觉-语言-动作模型(VLAs)等基于密集补丁特征的策略,但其计算开销巨大,继承了百亿参数视觉-语言模型(VLM)主干的全部成本。本文提出Patch Policy,一种轻量级架构扩展,使基于Transformer的策略可直接处理预训练的密集补丁标记,而无需全量VLM的计算负担。核心是块因果注意力掩码,在保持标准策略时间因果性的同时,允许每观测关注大量补丁标记与状态信息。Patch Policy 轻量、快速且高效,在四个仿真环境和三个真实世界任务套件中,相比采用最先进的全局池化表示的策略,相对性能提升40%;超越微调后的OpenVLA-OFT达18%,仅需其约0.7%的参数量。我们认为该方法为机器人社区提供了一条便捷路径,可持续利用视觉表征学习进展,同时不牺牲训练效率与推理速度,适用于高频率、反应式控制场景。视频展示见 https://patch-policy.github.io
原文摘要 · Abstract (English)
Pretrained dense visual features from Vision Transformers (ViTs) are powerful yet have been underutilized in robot learning. Modern robot policies either compress each observation into a single global token, or rely on visual backbones trained from scratch, sacrificing both fine-grained spatial detail and the benefits of large-scale visual pre-training. While there exist policies that do operate on dense patch features like large vision-language-action models (VLAs), they tend to be heavy and slow, inheriting the full cost of a billion-parameter vision-language model (VLM) backbone. We close this gap with Patch Policy, a minimal architectural extension that enables transformer-based policies to consume dense pre-trained patch tokens directly without the computational overhead of a full VLM. At its core is a block-causal attention mask that preserves the temporal causality of standard policies while letting the model attend over many patch tokens per observation, alongside other state information. Patch Policy is lightweight, fast, and highly effective. Across four simulated and three real-world environment suites, our method achieves a 40% relative improvement over policies using state-of-the-art global-pooled representations. Furthermore, it surpasses fine-tuned OpenVLA-OFT by 18% while using roughly 0.7% of the parameters. We believe Patch Policy provides a pipeline for the robotics community to readily leverage continuing progress in visual representation learning, without sacrificing the training efficiency or inference speed required for high-frequency, reactive control. Videos can be viewed at https://patch-policy.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。