VAT让视觉变压器全层级特征参与决策，机器人任务成功率超98%
VAT: Vision Action Transformer by Unlocking Full Representation of ViT
- 用所有层特征融合视觉与动作，不只依赖最后一层
- 在4个LIBERO任务上平均成功率98.15%，刷新纪录
- 适合做机器人模仿学习，尤其重视模型表征完整性的研究
在机器人学习中，视觉变换器（ViT）是主流的视觉感知方法，但多数方法仅使用最后一层特征，导致信息浪费。本文提出视觉动作变换器（VAT），基于ViT扩展出完整的特征层次结构，通过跨所有变换器层处理专用动作标记与视觉特征，实现感知与动作生成的深层渐进融合。在一系列模拟操作任务中，VAT在4个LIBERO基准测试上达到98.15%的平均成功率，显著优于OpenVLA-OFT等先前方法，确立了新基准。本工作不仅提供一种强大的模仿学习模型，更证明充分挖掘视觉模型的‘表征轨迹’对推进机器人策略的关键作用。项目代码已开源：https://github.com/sellerbubble/VAT。
原文摘要 · Abstract (English)
In robot learning, Vision Transformers (ViTs) are standard for visual perception, yet most methods discard valuable information by using only the final layer's features. We argue this provides an insufficient representation and propose the Vision Action Transformer (VAT), a novel architecture that is extended from ViT and unlocks the full feature hierarchy of ViT. VAT processes specialized action tokens with visual features across all transformer layers, enabling a deep and progressive fusion of perception and action generation. On a suite of simulated manipulation tasks, VAT achieves a 98.15\% average success rate across four LIBERO benchmarks, establishing a new state-of-the-art by outperforming prior methods like OpenVLA-OFT. Our work presents not only a powerful model for imitation learning but also demonstrates the critical importance of leveraging the complete ''representation trajectory'' of vision models to advance robotic policy. The GitHub URL for the project code is https://github.com/sellerbubble/VAT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。