解决3D场景跨域拓扑差异问题,提升多模态理解能力
DoReMi: Bridging 3D Domains via Topology-Aware Domain-Representation Mixture of Experts
- 引入拓扑感知的专家混合模型,结合空间上下文与路由不确定性动态分配专家
- 在ScanNet和S3DIS数据集上分别达到80.1%和77.2%的mIoU,优于现有方法
- 适合需要跨域3D理解的场景,如自动驾驶与室内建模
构建统一的3D场景理解模型长期受限于不同传感器模态间的显著拓扑差异。尽管混合专家(MoE)架构是实现通用理解的有效途径,但现有3D MoE网络常受语义驱动路由偏差影响,难以处理‘语义一致但拓扑异构’的跨域数据。为此,我们提出DoReMi(拓扑感知的域表示混合专家)。通过基于多属性(如拓扑与纹理变化)的自监督预训练分支,锚定跨域结构先验;在此基础上,设计域感知专家分支,包含两个核心机制:域空间引导路由(DSR),通过提取空间上下文实现对局部拓扑变化的敏锐感知;熵控制动态分配(EDA),通过量化路由不确定性动态调整激活专家数量,保障训练稳定性。二者协同使DoReMi实现通用特征提取与高度自适应专家分配的深度融合。大量实验涵盖室内外场景,在多个任务中验证其优越性:在ScanNet验证集上达80.1% mIoU,S3DIS上达77.2%,全面超越现有最先进方法。代码即将发布。
原文摘要 · Abstract (English)
Constructing a unified 3D scene understanding model has long been hindered by the significant topological discrepancies across different sensor modalities. While applying the Mixture-of-Experts (MoE) architecture is an effective approach to achieving universal understanding, we observe that existing 3D MoE networks often suffer from semantics-driven routing bias. This makes it challenging to address cross-domain data characterized by "semantic consistency yet topological heterogeneity." To overcome this challenge, we propose DoReMi (Topology-Aware Domain-Representation Mixture of Experts). Specifically, we introduce a self-supervised pre-training branch based on multi attributes, such as topological and texture variations, to anchor cross-domain structural priors. Building upon this, we design a domain-aware expert branch comprising two core mechanisms: Domain Spatial-Guided Routing (DSR), which achieves an acute perception of local topological variations by extracting spatial contexts, and Entropy-controlled Dynamic Allocation (EDA), which dynamically adjusts the number of activated experts by quantifying routing uncertainty to ensure training stability. Through the synergy of these dual branches, DoReMi achieves a deep integration of universal feature extraction and highly adaptive expert allocation. Extensive experiments across various tasks, encompassing both indoor and outdoor scenes, validate the superiority of DoReMi. It achieves 80.1% mIoU on the ScanNet validation set and 77.2% mIoU on S3DIS, comprehensively outperforming existing state-of-the-art methods. The code will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。