通过伪帧增强与体素网络,提升3D密集描述的准确性。
PVCap: Towards Accurate 3D Dense Captioning via PseudoCap and VoxelCapNet

- 用实例随机混合生成多样空间布局的伪帧,增强数据多样性。
- 在ScanRefer和Nr3D上分别提升11.41%和13.99%的[email protected]。
- 适合关注3D场景描述、空间关系建模的研究者。
3D密集描述是一项新兴的视觉-语言任务,旨在为3D场景中的每个物体生成描述性句子。现有方法存在两大局限:其一,常用全局刚性变换(如旋转)增强场景,但未改变空间布局,而多样空间布局对学习物体间空间关系至关重要;其二,多数工作聚焦于描述生成流程设计,却使用简单网络结构处理骨干和检测头,后者对提取丰富语义信息至关重要。本文提出PVCap,包含PseudoCap与VoxelCapNet。PseudoCap在数据集实例层面采用随机混合技术,生成大量具有多样化空间布局的伪帧,并通过师生框架获得伪描述标签,显著增加训练样本并提升模型环境描述能力。VoxelCapNet引入基于体素特征的鲁棒描述网络,适配体素架构的标题生成头。该模型可作为未来3D密集描述研究的有力基线。在ScanRefer和Nr3D两个主流基准上进行大量实验,方法在[email protected]指标上分别超越当前最优水平11.41%和13.99%。代码将公开。
原文摘要 · Abstract (English)
3D dense captioning, an emerging vision-language task, aims to generate descriptive sentences for each object in the 3D scene. Despite the impressive results achieved by previous methods, they suffer from two limitations. First, current research often employs global rigid transformations, such as rotation, to augment scenes without changing their spatial layouts. However, diverse spatial layouts are crucial for training a 3D dense captioning model to describe spatial relations between objects. Second, previous works mainly focus on the design of the caption generation pipeline while utilizing a simple network architecture for other components, i.e., backbone and detection head, which is crucial for extracting rich semantic information for captioning. In this paper, we propose PVCap to alleviate the aforementioned problems. Our PVCap consists of PseudoCap and VoxelCapNet. Specifically, PseudoCap employs a random mixing technique on instances within the dataset, generating numerous pseudo frames with diverse spatial layouts at the instance level. By utilizing a teacher-student framework, PseudoCap obtains pseudo caption labels for these pseudo frames. This data augmentation approach significantly increases the number of training samples and enhances the model's ability to describe the environment effectively. Regarding VoxelCapNet, we introduce a robust caption network that utilizes voxel features and adapts the caption head to the voxel-based network architecture. Our VoxelCapNet can serve as a competitive baseline for future research on 3D dense captioning. Extensive experiments are conducted on two prevalent benchmarks, i.e., ScanRefer and Nr3D. Notably, our method surpasses current state-of-the-art by 11.41% and 13.99% in [email protected], respectively. Codes will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。