OTTER通过语义对齐的视觉特征提取，实现零样本机器人操作泛化。
OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction
- 仅传递与语言指令相关的视觉特征，保留预训练语义对齐。
- 零样本测试中超越现有模型，在新物体和环境中表现优异。
- 无需微调视觉语言模型，适合需要稳定泛化的机器人任务。
视觉-语言-动作（VLA）模型旨在根据视觉观察和语言指令预测机器人动作。现有方法需微调预训练的视觉-语言模型，因视觉与语言特征独立输入下游策略，破坏了预训练的语义对齐。我们提出OTTER，一种新型VLA架构，通过显式的文本感知视觉特征提取利用现有对齐。OTTER不处理所有视觉特征，而是选择性提取与语言指令语义对齐的任务相关视觉特征传入策略Transformer。这使OTTER可保持预训练视觉-语言编码器冻结，从而保留并利用大规模预训练中的丰富语义理解，实现强大的零样本泛化能力。在仿真和真实世界实验中，OTTER显著优于现有VLA模型，展示了对新物体和环境的强零样本泛化能力。视频、代码、检查点及数据集：https://ottervla.github.io/。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models aim to predict robotic actions based on visual observations and language instructions. Existing approaches require fine-tuning pre-trained visionlanguage models (VLMs) as visual and language features are independently fed into downstream policies, degrading the pre-trained semantic alignments. We propose OTTER, a novel VLA architecture that leverages these existing alignments through explicit, text-aware visual feature extraction. Instead of processing all visual features, OTTER selectively extracts and passes only task-relevant visual features that are semantically aligned with the language instruction to the policy transformer. This allows OTTER to keep the pre-trained vision-language encoders frozen. Thereby, OTTER preserves and utilizes the rich semantic understanding learned from large-scale pre-training, enabling strong zero-shot generalization capabilities. In simulation and real-world experiments, OTTER significantly outperforms existing VLA models, demonstrating strong zeroshot generalization to novel objects and environments. Video, code, checkpoints, and dataset: https://ottervla.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。