提出PMT，用冻结视觉编码器实现图像视频分割的高速高精度
PMT: Plain Mask Transformer for Image and Video Segmentation with Frozen Vision Encoders
- 设计轻量级Transformer解码器，直接作用于冻结编码器特征
- 图像分割达顶尖性能，速度比现有冻结模型快3倍；视频分割媲美全微调方法
- 适合需要多任务共享编码器的高效部署场景
大规模预训练的视觉基础模型（VFMs）可让单一冻结编码器同时服务多个下游任务。近期基于VFMs的编码器-仅模型如EoMT和VidEoMT在图像与视频分割上实现了优异精度与极低延迟，但需微调编码器，牺牲了VFMs在大规模部署中多任务共享编码器的优势。为兼顾编码器-仅设计的简洁性与速度，同时保持冻结的视觉特征，我们提出纯掩码解码器（PMD），一种基于Transformer的快速分割解码器，运行于冻结的VFMs特征之上。由此构建的纯掩码变压器（PMT）保留了编码器-仅架构的简洁性与低延迟，且不改变编码器表示，可通用共享。该设计无缝适用于图像与视频分割，继承了编码器-仅框架的通用性。在标准图像分割基准上，PMT达到冻结编码器最先进的性能，速度提升约3倍；在视频分割上，其表现与全微调方法相当，且比现有最先进冻结编码器模型快达8倍。代码已开源。
原文摘要 · Abstract (English)
Vision Foundation Models (VFMs) pre-trained at scale enable a single frozen encoder to serve multiple downstream tasks simultaneously. Recent VFM-based encoder-only models for image and video segmentation, such as EoMT and VidEoMT, achieve competitive accuracy with remarkably low latency, yet they require finetuning the encoder, sacrificing the multi-task encoder sharing that makes VFMs practically attractive for large-scale deployment. To reconcile encoder-only simplicity and speed with frozen VFM features, we propose the Plain Mask Decoder (PMD), a fast Transformer-based segmentation decoder that operates on top of frozen VFM features. The resulting model, the Plain Mask Transformer (PMT), preserves the architectural simplicity and low latency of encoder-only designs while keeping the encoder representation unchanged and shareable. The design seamlessly applies to both image and video segmentation, inheriting the generality of the encoder-only framework. On standard image segmentation benchmarks, PMT matches the frozen-encoder state of the art while running up to ~3x faster. For video segmentation, it even performs on par with fully finetuned methods, while being up to 8x faster than state-of-the-art frozen-encoder models. Code: https://github.com/tue-mps/pmt.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。