arXiv:2605.07356cs.CV2026-05

统一2D与3D语义分割,提升跨域泛化能力。

UniD-Shift: Towards Unified Semantic Segmentation via Interpretable Share-Private Multimodal Decomposition

论文配图:UniD-Shift: Towards Unified Semantic Segmentation via Interpretable Share-Private Multimodal Decomposition
图 1 · 摘自论文原文
  • 分离共享与私有特征,实现跨模态语义对齐。
  • 在SemanticKITTI和nuScenes上准确率优于主流基线。
  • 适合自动驾驶、城市数字孪生等需要多模态融合的场景。

大规模3D点云的语义分割对自动驾驶和城市数字孪生至关重要。然而,激光雷达的稀疏采样模式以及图像观测中的视角依赖性几何失真,使跨模态对齐复杂化并阻碍稳定融合。受摄像头拍摄的2D图像是3D世界的表征这一事实启发,我们发现2D与3D分割所学特征具有部分共同语义,其余则保持模态特异性。基于此,提出统一的多模态框架,联合进行2D-3D语义分割。采用基于SAM的视觉编码器与基于SPTNet的几何编码器,提取互补的语义与几何表示。双模态特征被显式分解为共享与私有子空间:共享成分总结两域共有的语义因素,私有成分保留各模态特有属性。通过轻量级注意力融合模块将共享特征聚合为一致的跨模态表示,并引入正则化训练目标以确保语义对齐与子空间独立性。在SemanticKITTI和nuScenes基准上的实验表明,分割精度持续优于代表性多模态基线,同时具备竞争力的计算效率。在nuScenes USA-Singapore跨域评估中表现出稳定性能,证明强泛化能力。代码已公开于:https://github.com/shuaizhang69/UniD-Shift。

原文摘要 · Abstract (English)

Semantic segmentation of large-scale 3D point clouds is crucial for applications such as autonomous driving and urban digital twins. However, the sparse sampling pattern of LiDAR and the view-dependent geometric distortion in image observations complicate cross-modal alignment and hinder stable fusion. Inspired by the fact that 2D images captured by cameras are representations of the 3D world, we recognize that the features learned from 2D and 3D segmentation share some common semantics, while other aspects remain modality-specific. This insight motivates a unified multimodal framework for joint 2D-3D semantic segmentation. We combine a SAM-based vision encoder with a SPTNet-based geometric encoder to extract complementary semantic and geometric representations. The resulting features from both modalities are explicitly decomposed into shared and private subspaces, where the shared components summarize semantic factors common to both domains, and the private components preserve properties that are unique to each modality. A lightweight attention-based fusion module aggregates the shared features into a consistent cross-modal representation, and a regularized training objective ensures both semantic alignment and subspace independence. Experiments on the SemanticKITTI and nuScenes benchmarks demonstrate consistent improvements in segmentation accuracy over representative multimodal baselines, accompanied by competitive computational efficiency. Cross-domain evaluation on nuScenes USA-Singapore shows stable performance under distribution shifts, demonstrating strong generalization. The implementation code is publicly available at: https://github.com/shuaizhang69/UniD-Shift.

语义分割多模态融合3D感知跨域泛化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。