Multiformer通过混合分支融合实现深度感知视频全景分割新突破
Balancing Shared and Task-Specific Representations: A Hybrid Approach to Depth-Aware Video Panoptic Segmentation
- 在解码器中引入任务专用分支，再融合为共享表示
- 在Cityscapes和SemKITTI数据集上分别提升3.0和4.0 DVPQ点
- 适合研究多任务视觉理解与模型架构设计的学者
本文提出Multiformer，一种基于掩码变换器范式的深度感知视频全景分割（DVPS）新方法。该方法学习跨分割、单目深度估计和目标跟踪三个子任务的共享对象表征。与以往逐步精炼统一表征的统一方法不同，我们提出一种混合方法：在每个解码器块内使用任务专用分支，最终在块间接口处融合为共享表示。在Cityscapes-DVPS和SemKITTI-DVPS数据集上的大量实验表明，Multiformer在所有DVPS指标上均达到最先进性能，相比此前最佳方法分别提升了3.0和4.0 DVPQ点；使用ResNet-50主干时，深度估计精度也得到提升。该工作还为多任务解码器架构设计提供了重要洞见。
原文摘要 · Abstract (English)
In this work, we present Multiformer, a novel approach to depth-aware video panoptic segmentation (DVPS) based on the mask transformer paradigm. Our method learns object representations that are shared across segmentation, monocular depth estimation, and object tracking subtasks. In contrast to recent unified approaches that progressively refine a common object representation, we propose a hybrid method using task-specific branches within each decoder block, ultimately fusing them into a shared representation at the block interfaces. Extensive experiments on the Cityscapes-DVPS and SemKITTI-DVPS datasets demonstrate that Multiformer achieves state-of-the-art performance across all DVPS metrics, outperforming previous methods by substantial margins. With a ResNet-50 backbone, Multiformer surpasses the previous best result by 3.0 DVPQ points while also improving depth estimation accuracy. Using a Swin-B backbone, Multiformer further improves performance by 4.0 DVPQ points. Multiformer also provides valuable insights into the design of multi-task decoder architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。