arXiv:2608.02197cs.RO2026-08

让视觉模型专注关键区域,提升机器人操作的精准度。

Look Where It Matters: Adaptive Visual Refinement for Vision-Language-Action Models

论文配图:Look Where It Matters: Adaptive Visual Refinement for Vision-Language-Action Models
图 1 · 摘自论文原文
  • 引入可学习的注册令牌,专用于承载空间信息。
  • 在真实场景中成功率从46.5%提升至69.0%,平均提升4.2个百分点。
  • 仅在30%不确定步骤触发高分辨率重编码,计算开销增加1.4-1.6倍。

视觉语言动作模型的视觉表征在空间精确的机器人操作中仍不可靠。我们发现,这些模型的视觉编码器存在类似通用视觉变压器中的注意力伪影,并且这些伪影与后训练阶段习得的空间感知能力密切相关。当编码器学习物体位置、深度排序和局部几何等任务相关知识时,受限的全局令牌容量导致部分信息溢出至低信息量的图像块令牌中。为此,我们提出AtVLA框架,在视觉编码器中插入可学习的注册令牌。通过仅使用具身数据和原始动作目标进行端到端训练,这些注册令牌自发成为专门承载具身空间信息的载体,而其余块令牌则恢复清晰且空间忠实的注意力分布,对精确目标定位和细粒度接触至关重要。为进一步修复低分辨率观测中丢失的几何细节,AtVLA将注意力校正与置信度门控的局部精修结合:动作专家采样多个动作片段,根据其不一致程度估计不确定性;仅在不确定预测时,动作条件注意力滚动识别任务相关区域,该区域被裁剪、以高分辨率重新编码,并附加到缓存前缀中,实现精细化动作生成。在LIBERO、SimplerEnv及一个具有挑战性的单视角真实世界基准上,AtVLA将平均LIBERO成功率从94.2%提升至98.4%,真实世界成功率从46.5%提升至69.0%。裁剪操作在约30%的重规划步骤中触发,代表部署设置下总计算量仅增加1.4-1.6倍。

原文摘要 · Abstract (English)

Visual representations of VLA models remain unreliable for spatially precise robotic manipulation. We uncover that vision encoders in VLAs also exhibit attention artifacts previously documented in generic Vision Transformers, and further show that, in embodied policies, these artifacts are closely associated with spatial perception capabilities acquired during post-training. As the encoder learns task-relevant information such as object location, depth ordering, and local geometry, limited global-token capacity causes part of this information to spill into low-information patch tokens. We introduce AtVLA, a framework that inserts learnable register tokens into the visual encoder. Trained end-to-end using only embodied data and the original action objective, these registers emerge as dedicated carriers of embodied spatial information, while the remaining patch tokens recover clean and spatially faithful attention distributions crucial for precise target localization and fine-grained contact. Clean attention restores reliable localization, but cannot recover geometric details lost in low-resolution observations. AtVLA therefore couples attention rectification with uncertainty-gated local refinement. The action expert samples multiple action chunks and estimates uncertainty from their disagreement; only for uncertain predictions, action-conditioned attention rollout identifies the task-relevant region, which is cropped, re-encoded at high resolution, and appended to the cached prefix for refined action generation. Across LIBERO, SimplerEnv, and a challenging single-view real-world benchmark, AtVLA improves the average LIBERO success rate from 94.2% to 98.4% and real-world success from 46.5% to 69.0%. The cropping is triggered on approximately 30% of replanning steps, resulting in only 1.4-1.6x the total computation of the base model under the representative deployment setting.

机器人操作视觉注意空间感知具身智能

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。