用普通Transformer处理3D场景,仅靠数据增强就达到顶尖效果
Volume Transformer: Revisiting Vanilla Transformers for 3D Scene Understanding

- 将3D场景切分成体素块,用全连接自注意力+3D旋转位置编码
- 结合强数据增强和教师模型蒸馏,在小数据上实现超越主流方法
- 可直接替换现有3D分割流程,适合追求通用性和可扩展性的研究者
Transformer已成为深度学习的通用基础,但3D场景理解仍依赖具强领域先验的专用主干网络,导致该领域与更广泛的Transformer生态脱节,限制了跨领域研究成果迁移及软硬件优化红利。为此,我们提出体积Transformer(Volt),仅做少量修改即可将标准Transformer编码器用于3D场景。Volt将3D场景划分为体素块令牌,通过全局自注意力处理,并使用3D旋转位置编码注入位置信息。初步实验发现,直接在标准3D基准上训练Volt表现不佳,凸显当前3D监督数据规模有限。为此,我们设计了一套高效数据训练方案,包含强3D数据增强、正则化和卷积教师模型蒸馏,使Volt性能媲美最先进方法。随后通过多数据集联合训练扩大监督信号,发现Volt比领域专用3D主干更受益于数据规模增长,在多个室内与室外语义分割基准上取得新纪录。最后,在标准3D实例分割流程中作为即插即用主干,Volt也创下新高,彰显其作为简单、可扩展、通用的3D场景理解主干的巨大潜力。
原文摘要 · Abstract (English)
Transformers have become a common foundation across deep learning, yet 3D scene understanding still relies on specialized backbones with strong domain priors. This isolates the field from the broader Transformer ecosystem, limiting the transfer of research advances from other domains and the benefits of increasingly optimized software and hardware stacks. To bridge this gap, we propose the Volume Transformer (Volt), which adapts the vanilla Transformer encoder to 3D scenes with minimal modifications. Specifically, Volt partitions 3D scenes into volumetric patch tokens, processes them with full global self-attention, and injects positional information via 3D rotary positional embeddings (RoPE). Our initial experiments reveal that naively training Volt on standard 3D benchmarks leads to poor generalization, highlighting the limited scale of current 3D supervision. To overcome this, we introduce a data-efficient training recipe based on strong 3D augmentations, regularization, and distillation from a convolutional teacher, making Volt competitive with state-of-the-art methods. We then scale supervision through joint training on multiple datasets and show that Volt benefits more from increased scale than domain-specific 3D backbones, achieving state-of-the-art results on several indoor and outdoor semantic segmentation benchmarks. Finally, as a drop-in backbone in a standard 3D instance segmentation pipeline, Volt also sets a new state of the art, highlighting its potential as a simple, scalable, and general-purpose backbone for 3D scene understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。