大模型ViT可直接做图像分割，无需额外模块
Your ViT is Secretly an Image Segmentation Model
- 用大规模预训练让ViT自己学出分割所需特征
- 在相同精度下比现有方法快4倍，尤其适合ViT-L
- 证明扩大模型比加复杂结构更有效
视觉变换器（ViTs）在多种计算机视觉任务中表现出色且具备良好可扩展性。为将单尺度ViT用于图像分割，现有方法通常引入卷积适配器生成多尺度特征、像素解码器融合特征，并使用基于Transformer的解码器进行预测。本文发现，若模型足够大且预训练充分，这些任务特定组件所引入的归纳偏置可由ViT自身学习获得。基于此，我们提出仅编码器的掩码变换器（EoMT），直接复用纯ViT架构实现图像分割。在大规模模型与充分预训练下，EoMT达到与采用专用组件的最先进模型相当的分割精度，同时因架构简单而显著提速，例如在ViT-L上最快达4倍。在不同模型规模下，EoMT均展现出精度与速度的最佳平衡，表明计算资源应优先用于扩大模型本身而非增加结构复杂度。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) have shown remarkable performance and scalability across various computer vision tasks. To apply single-scale ViTs to image segmentation, existing methods adopt a convolutional adapter to generate multi-scale features, a pixel decoder to fuse these features, and a Transformer decoder that uses the fused features to make predictions. In this paper, we show that the inductive biases introduced by these task-specific components can instead be learned by the ViT itself, given sufficiently large models and extensive pre-training. Based on these findings, we introduce the Encoder-only Mask Transformer (EoMT), which repurposes the plain ViT architecture to conduct image segmentation. With large-scale models and pre-training, EoMT obtains a segmentation accuracy similar to state-of-the-art models that use task-specific components. At the same time, EoMT is significantly faster than these methods due to its architectural simplicity, e.g., up to 4x faster with ViT-L. Across a range of model sizes, EoMT demonstrates an optimal balance between segmentation accuracy and prediction speed, suggesting that compute resources are better spent on scaling the ViT itself rather than adding architectural complexity. Code: https://www.tue-mps.org/eomt/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。