用ViT做视频分割，不用专门追踪模块也能快10倍
VidEoMT: Your ViT is Secretly Also a Video Segmentation Model
- 纯编码器结构，靠查询传播实现时序建模
- 在YouTube-VOS上达81.4% mIoU，速度超160 FPS
- 适合追求高速高精度视频分割的开发者
现有在线视频分割模型通常结合帧级分割器与复杂专用追踪模块，虽有效但增加架构复杂度和计算开销。近期研究表明，经过充分容量扩展和大规模预训练的纯视觉变换器（ViT）编码器，无需专用模块即可实现精准图像分割。受此启发，我们提出视频编码器仅需掩码变换器（VidEoMT），一种无需专用追踪模块的简单编码器型视频分割模型。为在纯编码器ViT中实现时序建模，VidEoMT引入轻量级查询传播机制，通过复用前一帧的查询传递信息。为兼顾对新内容的适应性，采用查询融合策略，将传播查询与一组与时序无关的可学习查询结合。结果表明，VidEoMT在不增加复杂度的情况下获得类似追踪器的效果，在准确率上表现优异，同时速度提升5至10倍，使用ViT-L主干网络时最高可达160 FPS。代码已开源。
原文摘要 · Abstract (English)
Existing online video segmentation models typically combine a per-frame segmenter with complex specialized tracking modules. While effective, these modules introduce significant architectural complexity and computational overhead. Recent studies suggest that plain Vision Transformer (ViT) encoders, when scaled with sufficient capacity and large-scale pre-training, can conduct accurate image segmentation without requiring specialized modules. Motivated by this observation, we propose the Video Encoder-only Mask Transformer (VidEoMT), a simple encoder-only video segmentation model that eliminates the need for dedicated tracking modules. To enable temporal modeling in an encoder-only ViT, VidEoMT introduces a lightweight query propagation mechanism that carries information across frames by reusing queries from the previous frame. To balance this with adaptability to new content, it employs a query fusion strategy that combines the propagated queries with a set of temporally-agnostic learned queries. As a result, VidEoMT attains the benefits of a tracker without added complexity, achieving competitive accuracy while being 5x-10x faster, running at up to 160 FPS with a ViT-L backbone. Code: https://www.tue-mps.org/videomt/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。