机器人靠记忆决定该看什么,而非如何走,提升感知效率。
Memory for Attention: Language-Conditioned Re-Perception with a Vision--Language--Motion Map
- 用记忆判断何时重看物体,替代盲目感知
- 关键物体重看频率提升1.6倍,任务失误减少
- 适合需要语言指令跟踪目标的机器人场景
携带持续行为标注地图的机器人面临两个规划问题:空间导航与资源分配。在28个AI2-THOR场景中,基于视觉-语言-动作地图(VLMM)的行为感知规划器虽使计划时间成本降低约35%,但闭环执行时实际收益仅约4%;按需调用的视觉-语言模型(VLM)表现相当。而针对感知预算有限时应重看何物的问题,将重感知建模为注意力决策,发现地图的记忆(如变化历史或最近观测时间)能生成最优重看策略,媲美理想情况下的预言者(oracle),而无记忆的VLM先验效果差。该策略使感知预算集中于重要物体,其重看频率达平均值的1.6倍;下游取物任务显示浪费行程减少。增益随实例间异质性增加,符合柯西-施瓦茨界预测,等于√λ的方差。使用真实CLIP先验时,性能提升达21%-26%。当任务受语言条件约束时,VLMM能以开集方式定位并追踪指定对象,优于仅依赖相关性或最近时间的基线(+2.5%),也优于按需VLM(+8.9%)——说明动态信息与语言理解缺一不可。地图的核心价值不在于指导行走,而在于指导注意。
原文摘要 · Abstract (English)
A robot carrying a persistent, behavior-annotated map faces two planning questions, and its memory answers only one well. The \emph{spatial-navigation} question -- how to walk around a room -- we address first and report a negative: building on Vision--Language--Motion Maps (VLMM), a behavior-aware planner cost cuts a planning-time objective by $\sim$35\% over 28 AI2-THOR scenes, but under closed-loop execution the real benefit nearly vanishes ($\sim$4\%) and an on-demand vision--language model (VLM) does as well. The \emph{resource-allocation} question differs: under a limited perception budget, what should the robot re-observe now to keep its map fresh? Framing re-perception as this attention decision, we show a persistent map's memory (change-history, or even just recency of last sighting) yields the best schedule (held-out), matching an oracle, while the memoryless VLM prior is poor. Because the schedule reallocates budget toward what matters, memory's benefit concentrates on the important objects ($\sim$1.6$\times$ the mean), and a downstream fetch task confirms fewer wasted trips; the gain grows with per-instance heterogeneity exactly as a Cauchy--Schwarz bound predicts -- it equals $\mathrm{Var}(\sqrtλ)$, the variance of root-volatility. With a real CLIP prior on rendered objects the advantage is $+21$--$26\%$. The map's distinctive value appears when the task is \emph{language-conditioned}: told what to track, VLMM grounds the relevant objects (open-vocabulary) and tracks their change (memory), beating even a strong relevance-weighted recency baseline ($+2.5\%$) -- so its motion channel adds value beyond a last-seen timestamp -- and an on-demand VLM ($+8.9\%$); neither language nor dynamics alone suffices. The map earns its keep not by telling the robot how to walk around a room, but by telling it what to pay attention to.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。