用单个上下文标记提升机器人多帧视觉理解效率
ContextVLA: Vision-Language-Action Model with Amortized Multi-Frame Context
- 将多帧图像压缩为一个上下文标记，高效利用时间信息
- 相比单帧模型，任务成功率显著提升，训练推理更快
- 适合需要实时决策的机器人控制场景
在部分可观测的机器人任务中，利用时间上下文至关重要。然而，以往的行为克隆研究在使用多帧观测时表现不一。本文提出ContextVLA，一种基于视觉-语言-动作模型（VLA）的策略模型，能更有效地利用多帧观测生成动作。我们发现，视觉-语言模型（VLM）具备内在的时间理解能力，可从多帧输入中提取更有意义的上下文。但视频输入的高维性带来了巨大的计算开销。为此，ContextVLA将历史观测压缩为单一上下文标记，使策略能高效利用时间上下文进行动作生成。实验表明，ContextVLA在性能上持续优于单帧VLA，达到全多帧训练的效果，同时显著降低训练和推理时间。
原文摘要 · Abstract (English)
Leveraging temporal context is crucial for success in partially observable robotic tasks. However, prior work in behavior cloning has demonstrated inconsistent performance gains when using multi-frame observations. In this paper, we introduce ContextVLA, a policy model that robustly improves robotic task performance by effectively leveraging multi-frame observations. Our approach is motivated by the key observation that Vision-Language-Action models (VLA), i.e., policy models built upon a Vision-Language Model (VLM), more effectively utilize multi-frame observations for action generation. This suggests that VLMs' inherent temporal understanding capability enables them to extract more meaningful context from multi-frame observations. However, the high dimensionality of video inputs introduces significant computational overhead, making VLA training and inference inefficient. To address this, ContextVLA compresses past observations into a single context token, allowing the policy to efficiently leverage temporal context for action generation. Our experiments show that ContextVLA consistently improves over single-frame VLAs and achieves the benefits of full multi-frame training but with reduced training and inference times.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。