让3D场景描述更细腻:同时捕捉物体整体与细节特征。
ExCap3D: Expressive 3D Scene Understanding via Object Captioning with Varying Detail
- 分层生成描述:先整体后局部,整合物体与部件信息
- 在ScanNet++上生成19万条多粒度描述,提升质量124%以上
- 适合需要精细理解3D环境的机器人、VR/AR应用
生成3D室内场景中物体的文本描述是具身理解的重要基础。现有方法仅以单一粒度描述物体,难以捕捉纹理、材质和部件形状等细粒度信息。本文提出表达性3D描述任务:给定3D场景,同时生成物体的高层描述和部件的低层属性描述。为此,我们提出ExCap3D模型,输入3D扫描数据,对每个检测到的物体,生成包含部件集体描述的细粒度文本,并基于部件描述生成对象级描述。模型通过语义一致性和隐空间文本相似性约束提升生成质量。为支持该任务,我们利用视觉-语言模型(VLM)构建了ExCap3D数据集,包含947个室内场景中34,000个3D物体的190,000条多粒度文本描述。实验表明,ExCap3D生成的物体和部件级描述质量显著优于现有方法,分别在Cider评分上提升17%和124%。代码、数据集和模型将公开发布。
原文摘要 · Abstract (English)
Generating text descriptions of objects in 3D indoor scenes is an important building block of embodied understanding. Existing methods do this by describing objects at a single level of detail, which often does not capture fine-grained details such as varying textures, materials, and shapes of the parts of objects. We propose the task of expressive 3D captioning: given an input 3D scene, describe objects at multiple levels of detail: a high-level object description, and a low-level description of the properties of its parts. To produce such captions, we present ExCap3D, an expressive 3D captioning model which takes as input a 3D scan, and for each detected object in the scan, generates a fine-grained collective description of the parts of the object, along with an object-level description conditioned on the part-level description. We design ExCap3D to encourage semantic consistency between the generated text descriptions, as well as textual similarity in the latent space, to further increase the quality of the generated captions. To enable this task, we generated the ExCap3D Dataset by leveraging a visual-language model (VLM) for multi-view captioning. The ExCap3D Dataset contains captions on the ScanNet++ dataset with varying levels of detail, comprising 190k text descriptions of 34k 3D objects in 947 indoor scenes. Our experiments show that the object- and part-level of detail captions generated by ExCap3D are of higher quality than those produced by state-of-the-art methods, with a Cider score improvement of 17% and 124% for object- and part-level details respectively. Our code, dataset and models will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。