用空间规则提升3D指代分割精度，解决误分割问题
RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation
- 仅用目标实例的空间信息做监督，引导模型理解文本描述的空间关系
- 在ScanRefer上提升mIoU 5.1点，空间模糊描述下更稳定
- 适合需要精准空间推理的3D视觉任务，如机器人导航
3D指代表达分割（3D-RES）旨在通过关联文本描述与点云来分割3D物体。然而，传统方法常因忽略实例的空间信息而出现过分割或错分割。本文提出规则引导的空间感知网络（RG-SAN），仅利用目标实例的空间信息进行监督，使网络能准确刻画文本中所有实体间的空间关系，从而增强推理能力。RG-SAN包含文本驱动定位模块（TLM）和规则引导弱监督策略（RWS）。TLM先定位所有提及实例，并迭代优化其位置；RWS策略基于依赖树规则，精准指导核心实例定位，尽管仅目标对象有标注位置。在ScanRefer基准上的大量实验表明，RG-SAN不仅创下新性能纪录（mIoU提升5.1点），且在处理空间模糊描述时表现出更强鲁棒性。代码已开源：https://github.com/sosppxo/RG-SAN。
原文摘要 · Abstract (English)
3D Referring Expression Segmentation (3D-RES) aims to segment 3D objects by correlating referring expressions with point clouds. However, traditional approaches frequently encounter issues like over-segmentation or mis-segmentation, due to insufficient emphasis on spatial information of instances. In this paper, we introduce a Rule-Guided Spatial Awareness Network (RG-SAN) by utilizing solely the spatial information of the target instance for supervision. This approach enables the network to accurately depict the spatial relationships among all entities described in the text, thus enhancing the reasoning capabilities. The RG-SAN consists of the Text-driven Localization Module (TLM) and the Rule-guided Weak Supervision (RWS) strategy. The TLM initially locates all mentioned instances and iteratively refines their positional information. The RWS strategy, acknowledging that only target objects have supervised positional information, employs dependency tree rules to precisely guide the core instance's positioning. Extensive testing on the ScanRefer benchmark has shown that RG-SAN not only establishes new performance benchmarks, with an mIoU increase of 5.1 points, but also exhibits significant improvements in robustness when processing descriptions with spatial ambiguity. All codes are available at https://github.com/sosppxo/RG-SAN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。