arXiv:2503.03190cs.CV2025-03CVPR被引 15

融合多视角图像与点云,提升3D问答的鲁棒性。

DSPNet: Dual-vision Scene Perception for Robust 3D Question Answering

  • 通过文本引导选择关键图像视角,增强语义匹配
  • 自适应融合图像与点云特征,缓解姿态噪声影响
  • 适合需要跨模态理解的3D场景问答任务

3D问答(3D QA)要求模型全面理解由文本描述的三维场景,推理周围环境并回答问题。现有方法通常依赖纯点云的全局感知,忽视了多视角图像中的丰富局部纹理细节。此外,由于相机位姿固有噪声和复杂遮挡,点云与多视角图像对齐时存在显著特征退化和鲁棒性下降问题。本文提出双视觉场景感知网络(DSPNet),全面整合多视角图像与点云特征,以提升3D QA的鲁棒性。提出的文本引导多视角融合(TGMF)模块优先选择与文本语义匹配度高的图像视角;自适应双视觉感知(ADVP)模块则动态融合反投影的多视角图像与点云特征,增强3D场景理解。此外,多模态上下文引导推理(MCGR)模块通过融合视觉与语言模态的上下文信息,实现更稳健的推理。在SQA3D和ScanQA数据集上的实验结果表明,DSPNet性能优于现有方法。代码将公开于https://github.com/LZ-CH/DSPNet。

原文摘要 · Abstract (English)

3D Question Answering (3D QA) requires the model to comprehensively understand its situated 3D scene described by the text, then reason about its surrounding environment and answer a question under that situation. However, existing methods usually rely on global scene perception from pure 3D point clouds and overlook the importance of rich local texture details from multi-view images. Moreover, due to the inherent noise in camera poses and complex occlusions, there exists significant feature degradation and reduced feature robustness problems when aligning 3D point cloud with multi-view images. In this paper, we propose a Dual-vision Scene Perception Network (DSPNet), to comprehensively integrate multi-view and point cloud features to improve robustness in 3D QA. Our Text-guided Multi-view Fusion (TGMF) module prioritizes image views that closely match the semantic content of the text. To adaptively fuse back-projected multi-view images with point cloud features, we design the Adaptive Dual-vision Perception (ADVP) module, enhancing 3D scene comprehension. Additionally, our Multimodal Context-guided Reasoning (MCGR) module facilitates robust reasoning by integrating contextual information across visual and linguistic modalities. Experimental results on SQA3D and ScanQA datasets demonstrate the superiority of our DSPNet. Codes will be available at https://github.com/LZ-CH/DSPNet.

3D问答多模态融合点云理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。