无需训练,通过记忆实现3D物体操作区域精准定位。
Grounding by Remembering: Cross-Scene and In-Scene Memory for 3D Functional Affordances

- 构建跨场景与场景内两级记忆,辅助语言模型精确定位操作区域。
- 在SceneFun3D上比现有无训练方法提升3.23~3.7的检测准确率。
- 适合需要零样本、高精度交互区域定位的研究与应用。
功能性操作区域的定位不仅需识别物体,还需精确定位可交互区域,如拉手或按钮。这对无训练视觉-语言模型是挑战,因可操作区域小、视觉模糊且同类物体重复出现。本文提出AFFORDMEM框架,通过两级记忆机制实现3D功能操作区域的定位:一是跨场景操作记忆,维护类别级记忆库,以带操作区域标记的RGB图像为参考,在查询时召回最具信息量的例子,引导冻结的视觉语言模型定位细粒度操作区;二是场景内空间记忆,将候选实例及其3D空间关系组织成结构化场景图,使语言模型能处理远距离或未观测目标的指代,如“从上数第二个把手”。该方法无需模型微调或目标场景标注,使用源场景构建的可复用记忆库。在SceneFun3D数据集上,相比现有无训练最优方法,本方法在Split 0和Split 1上分别提升AP50达3.23和3.7。消融实验表明,跨场景记忆提升细粒度定位,场景内空间记忆对空间描述类查询贡献更大。
原文摘要 · Abstract (English)
Functional affordance grounding requires more than recognizing an object: an agent must localize the specific region that supports an interaction, such as the handle to pull or the button to press. This is difficult for training-free vision-language pipelines because actionable regions are often small, visually ambiguous, and repeated across multiple same-category instances in a scene. We propose AFFORDMEM, a framework that grounds 3D functional affordances by remembering geometry at two levels. The first is cross-scene affordance memory: the agent maintains a category-level memory bank of RGB images with affordance regions rendered as overlays, and recalls the most informative examples at query time to guide a frozen VLM toward small operable subregions that text-only prompting consistently misses. The second is in-scene spatial memory: as the agent processes the scene, it organizes candidate instances and their 3D spatial relations into a structured scene graph, enabling the language model to resolve references over distant or currently unobserved candidates such as "the second handle from the top." AFFORDMEM requires no model fine-tuning and no target-scene annotation, using a reusable memory bank built from source scenes. On SceneFun3D, our method improves AP50 over the prior training-free state of the art by 3.23 on Split 0 and 3.7 on Split 1. Ablation studies support complementary benefits: cross-scene affordance memory improves fine-grained localization, while in-scene spatial memory provides the larger gain on spatially qualified queries. The project homepage is available at the project page.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。