让视觉语言动作模型在不确定时自动调用视觉信息,无需额外训练。
UAOR: Uncertainty-aware Observation Reinjection for Vision-Language-Action Models
- 基于语言模型的不确定性判断,动态注入观察信息到下一层。
- 在仿真和真实任务中提升多种模型性能,且无额外计算开销。
- 无需额外数据或模块,可直接插入现有机器人系统使用。
视觉-语言-动作(VLA)模型利用预训练视觉-语言模型作为骨干,将图像和指令映射为动作,在通用机器人操作中展现出巨大潜力。现有方法常引入深度图、点云等额外观测信号或物体检测器等辅助模块以提升精度与可靠性,但需昂贵的数据采集和额外训练。受语言模型中前馈网络(FFN)可充当“键值记忆”的启发,本文提出不确定性感知的观测重注(UAOR),一种无需训练、即插即用的模块。当当前语言模型层表现出高不确定性(以动作熵衡量)时,该模块通过注意力检索,将关键观测信息注入下一层的FFN中,直接增强高不确定性层的隐藏状态,从而生成更准确可靠的行动。大量实验表明,该方法在多种仿真与真实任务中持续提升不同VLA模型表现,且几乎不增加计算负担。重要的是,它无需额外观测信号或模块,可作为通用实用组件嵌入现有VLA流水线。项目主页:https://uaor.jiabingyang.cn。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models leverage pretrained Vision-Language Models (VLMs) as backbones to map images and instructions to actions, demonstrating remarkable potential for generalizable robotic manipulation. To enhance performance, existing methods often incorporate extra observation cues (e.g., depth maps, point clouds) or auxiliary modules (e.g., object detectors, encoders) to enable more precise and reliable task execution, yet these typically require costly data collection and additional training. Inspired by the finding that Feed-Forward Network (FFN) in language models can act as "key-value memory", we propose Uncertainty-aware Observation Reinjection (UAOR), an effective, training-free and plug-and-play module for VLA models. Specifically, when the current language model layer exhibits high uncertainty, measured by Action Entropy, it reinjects key observation information into the next layer's Feed-Forward Network (FFN) through attention retrieval. This mechanism directly augments the hidden states with observation evidence at high-uncertainty layers, enabling more accurate and reliable action generation. Comprehensive experiments show that our method consistently improves diverse VLA models across simulation and real-world tasks with minimal overhead. Notably, UAOR eliminates the need for additional observation cues or modules, making it a versatile and practical plug-in for existing VLA pipelines. The project page is at https://uaor.jiabingyang.cn.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。