让机器人通过关系位置记忆精准找到特定物品，支持复杂语言指令。
FARM: Find Anything using Relational Spatial Memory
- 构建实时更新的物体级关系空间记忆，融合几何、视觉语言特征。
- 在67个场景中，召回率比之前方法提升164%（Recall@5）和224%（Recall@10）。
- 适合需要精准定位物体的机器人应用，如家庭服务与仓储巡检。
在家庭、仓库等物体密集环境中，机器人需具备按需检索特定物体实例的记忆系统。仅靠物体层级记忆常不足：场景中存在多个可能匹配的物体，用户常通过目标与地标及周围物体的关系来指认（例如“挂在飞镖板下方、海报左侧的高灯”），因此需要支持语义、外观和空间谓词的关联式空间记忆。为此，我们提出FARM（Find Anything using Relational Spatial Memory），可在5-10 Hz下实时构建紧凑、开放词汇、基于物体的记忆系统，包含几何信息、视觉-语言描述符和视角证据。查询时，FARM利用视觉语言模型（VLMs）解析查询并评分视觉证据，同时通过物体符号和关系谓词显式地建模空间约束。这种结构化使用VLM的方式，使检索准确性和鲁棒性优于端到端推理帧历史或场景图上下文的方法。在覆盖67个室内外场景（面积15至15,000平方米）、共4.4万条语言查询的实验中，FARM相较先前方法，Recall@5提升164%，Recall@10提升224%；最终的VLM重排序阶段进一步将Accuracy@1提升35%，且系统运行在实时条件下。我们还展示了在四足机器人上使用机载传感器与计算资源的闭环部署效果。
原文摘要 · Abstract (English)
Robots operating in homes, warehouses, and other object-rich environments need memory systems that can find specific object instances on demand. Object-level memory alone is often insufficient: scenes contain many plausibly matching objects, and users refer to the target through relations to landmarks and surrounding objects (e.g. ``the tall lamp below the dartboard and to the left of the poster''), demanding a relational spatial memory that supports retrieval through semantic, appearance, and spatial predicates over objects. To achieve this, we present FARM (Find Anything using Relational Spatial Memory), which builds, in real time at 5-10 Hz, a compact, open-vocabulary, object-level memory with geometry, visual-language descriptors, and viewpoint evidence. At query time, FARM uses VLMs to parse the query and score visual evidence, while grounding spatial constraints explicitly through object symbols and relational predicates. This structured use of VLMs enables more accurate and robust retrieval than end-to-end reasoning over frame histories or scene-graph context. In experiments on 44k language queries spanning 67 indoor and outdoor scenes, ranging from 15 to 15,000 m^2, FARM improves Recall@5 and Recall@10 over prior methods by 164% and 224%, and a final VLM reranking stage improves Accuracy@1 by 35%, while running in real time. We further demonstrate closed-loop deployment on a quadrupedal robot using onboard sensors and compute.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。